fc관련 log 분석 도움 부탁드립니다.

[김천기]

아래는 ioscan -fnC fc, fcmsutil, event.log 내용입니다. 정확히 어떤 조치를 취해야 할지 몰라서 고수님들께 도움 요청 드립니다. 도움 부탁드립니다.

kospi02# ioscan -fnC fc
Class I H/W Path Driver S/W State H/W Type Description
===================================================================
fc 0 0/0/1/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd0
fc 1 0/0/1/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd1
fc 2 0/0/2/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd2
fc 3 0/0/2/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd3
fc 4 0/0/8/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd4
fc 5 0/0/8/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd5
fc 6 0/0/10/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd6
fc 7 0/0/10/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd7
fc 8 0/0/12/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd8
fc 9 0/0/12/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd9
kospi02# fcmsutil /dev/fcd0

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000697015
N_Port Port World Wide Name = 0x50060b0000697014
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/1/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = YES
Driver-Firmware Dump Timestamp = Sat Oct 10 10:49:46 2009
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000697017
N_Port Port World Wide Name = 0x50060b0000697016
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/1/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd2

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x027000
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000696e81
N_Port Port World Wide Name = 0x50060b0000696e80
Switch Port World Wide Name = 0x207000051e362b5a
Switch Node World Wide Name = 0x100000051e362b5a
Driver state = ONLINE
Hardware Path is = 0/0/2/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd3

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000696e83
N_Port Port World Wide Name = 0x50060b0000696e82
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/2/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd4

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x018d00
Previous N_Port_id is = 0x018d00
N_Port Node World Wide Name = 0x50060b0000696f69
N_Port Port World Wide Name = 0x50060b0000696f68
Switch Port World Wide Name = 0x208d00051e362de4
Switch Node World Wide Name = 0x100000051e362de4
Driver state = ONLINE
Hardware Path is = 0/0/8/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = YES
Driver-Firmware Dump Timestamp = Sun Oct 4 23:02:32 2009
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd5

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000696f6b
N_Port Port World Wide Name = 0x50060b0000696f6a
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/8/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd6

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x017000
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000696e41
N_Port Port World Wide Name = 0x50060b0000696e40
Switch Port World Wide Name = 0x207000051e362de4
Switch Node World Wide Name = 0x100000051e362de4
Driver state = ONLINE
Hardware Path is = 0/0/10/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd7

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000696e43
N_Port Port World Wide Name = 0x50060b0000696e42
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/10/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd8

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 266 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b000068aaed
N_Port Port World Wide Name = 0x50060b000068aaec
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/12/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02# fcmsutil /dev/fcd9

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012d7
PCI Mode = PCI-X 266 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b000068aaef
N_Port Port World Wide Name = 0x50060b000068aaee
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/12/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17

kospi02 - check
>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Sat Oct 10 10:49:46 2009

kospi02 sent Event Monitor notification information:

/adapters/events/ql_adapter/0_0_1_1_0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Sat Oct 10 10:49:46 2009
Severity............: CRITICAL
Monitor.............: dm_ql_adapter
Event #.............: 45
System..............: kospi02

Summary:
Adapter at hardware path 0/0/1/1/0 : FCD driver received notification of
RISC Parity Error



Description of Error:


lbolt value: 706317478

fcd driver received notification of RISC Parity Error.


Probable Cause / Recommended Action:

Fibre Channel RISC controller chip detected a internal data and/or
instruction queue parity error.

fcd driver will try to take a driver and firmware dump to diagnose this
problem. Please save this dump for Hewlett Packard to analyze.


Additional Event Data:
System IP Address...: 10.223.50.12
Event Id............: 0x4acfe83a00000000
Monitor Version.....: B.02.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_dm_ql_adapter.clcfg
Client Configuration File Version...: A.04.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
0x4acfe83a00000000
Additional System Data:
System Model Number.............: ia64 hp server rx8640
OS Version......................: B.11.23
EMS Version.....................: A.04.20
STM Version.....................: C.60.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/dm_ql_adapter.htm#45

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
Physical Device Path....: 0/0/1/1/0

I/O Log Event Data:

Driver Status Code..................: 0x0000002D
Length of Logged Hardware Status....: 0 bytes.
Offset to Logged Manager Information: 0 bytes.
Manager-Specific Information:

Raw data from FCMS Adapter driver:
00000001 2A198CA6 00000000 00000001 2F75782F 636F7265 2F697375 2F464344
2F6B6572 6E2F7372 632F636F 6D6D6F6E 2F777369 6F2F6663 645F6973 722E63





>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Sat Oct 10 10:49:46 2009

kospi02 sent Event Monitor notification information:

/adapters/events/ql_adapter/0_0_1_1_0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Sat Oct 10 10:49:46 2009
Severity............: CRITICAL
Monitor.............: dm_ql_adapter
Event #.............: 29
System..............: kospi02

Summary:
Adapter at hardware path 0/0/1/1/0 : Fibre Channel Driver received a Fatal
Error Notification.



Description of Error:

lbolt value: 706317478

Fibre Channel Driver received a Fatal Error Notification.


Probable Cause / Recommended Action:

A possible cause is a Mailbox Command time out.
Please see Preceding Error messages in the log files to isolate
the cause. The Fibre Channel Driver will now perform fatal
error handling.

If there was no unretrieved dump for the adapter port, the driver will
have collected a new dump for the adapter port. This dump should be
retrieved as soon as possible using the command:
/opt/fcms/bin/fcmsutil dump_saved_state >

Please save the dump for HP analysis.

If auto-recovery is turned on, the driver will automatically
re-initialize the adapter port.

If auto-recovery is turned off, the adapter port will not automatically
be re-initialized. In this situation, please use the following command
to re-initialize the adapter port:
/opt/fcms/bin/fcmsutil reset


Additional Event Data:
System IP Address...: 10.223.50.12
Event Id............: 0x4acfe83a00000004
Monitor Version.....: B.02.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_dm_ql_adapter.clcfg
Client Configuration File Version...: A.04.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):

0x4acfe83a00000001
Additional System Data:
System Model Number.............: ia64 hp server rx8640
OS Version......................: B.11.23
EMS Version.....................: A.04.20
STM Version.....................: C.60.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/dm_ql_adapter.htm#29

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
Physical Device Path....: 0/0/1/1/0

I/O Log Event Data:

Driver Status Code..................: 0x0000001D
Length of Logged Hardware Status....: 0 bytes.
Offset to Logged Manager Information: 0 bytes.
Length of Logged Manager Information: 67 bytes.

Manager-Specific Information:

Raw data from FCMS Adapter driver:
00000001 2A198CA6 00000000 00000001 2F75782F 636F7265 2F697375 2F464344
2F6B6572 6E2F7372 632F636F 6D6D6F6E 2F777369 6F2F6663 645F6368 616E6E65
6C2E63

[김병수]

fc card에 연결되는 장비가 어떤것인지 궁금하네요.

기본적으로 서버에 설치된 fc card는 CLAIMED되어 정상설치되어 있구요.

fcmautil로 보시면 topology에 문제가 있어 보입니다.
연결하려는 대상장비에 따라 topology를 변경하여 보시기 바랍니다.

그럼~~~

[김천기]

김병수님 답변 주셔서 감사합니다.
ioscan으로 보면 아래와 같이 나오는데 좀 이상합니다. 원인을 알고 싶습니다. 도움 부탁드립니다.
cospi02# ioscan -fnH 0/0/1/1/0
Class I H/W Path Driver S/W State H/W Type Description
==================================================================
fc 0 0/0/1/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd0
cospi02# ioscan -fnH 0/0/1/1
Class I H/W Path Driver S/W State H/W Type Description
==================================================================
fc 0 0/0/1/1/0 fcd NO_HW INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd0
fc 1 0/0/1/1/1 fcd NO_HW INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd1
cospi02# ioscan -fnH 0/0/1
Class I H/W Path Driver S/W State H/W Type Description
==================================================================
ba 1 0/0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (12ee)
fc 0 0/0/1/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd0
fc 1 0/0/1/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd1
cospi02#

[김병수]

ioscan 시 NO_HW 때문이라면 ..

서버모델이 무엇인지요??

알려주시기 바랍니다.

[김천기]

ia64 hp server rx8640 입니다.

[김병수]

서버가 rx8640 이라면 AB379 card의 제약이 없습니다.

혹시모르니 fc card의 driver를 upgrade하여 보시기 바랍니다.

그래도 않될시에는 HP에 call을 하여 보심이..

rx8640장비는 HP에서 direct 지원하는 model인데...

그럼~~~

[김천기]

답변주셔서 감사합니다. ^^

[monoworld_1]

ioscan 시 path depth 에 따라 no_hw로 나오는 경우 있습니다.
동일시스템에서 다른 패스 잡고 해보셔도 동일한 결과 나올겁니다.
(ioscan -fnH 0/1/1/1 이런식으로 다른패스 )
OS에서 논리적으로 인식할수있는 디바이스가 없어서 그런거 같은데 여튼 다른 시스템에서도 이런 현상 있습니다.
하위 패스 잡고 ioscan 했을때 정상으로 나오면 문제 없을겁니다. 아님 ioscan -fn해서 no_hw 안떨어지면 정상이에요.

그리고 fcmsutil 결과만으로는 0/0/1/1/0 쪽 포트로 FC케이블 연결 안되어 있는것 같은데 연결 되어 있는 상태에서 로그 생성되었다면 문제 있지만, 연결 안되거나, 연결된것을 제거한 상태에서 생성된 로그라면 특별히 문제될거 없습니다.

김병수님 말대로 만든애들이 젤 잘알듯 싶습니다.