Fibre Channel Driver received a Fatal Error Notification

[Balaji Borlepawar]

Hi All,

One of my HP server rx7640 giving problem of mounting volumes after reboot throwing below messages,
1/0/6/1/0: Fibre Channel Driver received a Fatal Error Notification.
A possible cause is a Mailbox Command time out.
Please see Preceding Error messages to isolate the cause.
Fibre Channel Driver will now perform fatal error handling.....

Root check done....

/sbin/ioinitrc:
/dev/vg00/lvol1:file system is clean - log replay is not required
/sbin/krs_sysinit:
* The module 'rng' has been loaded.

insf: Installing special files for ipmi instance 0 address 1/250/1

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" is already active on this system.
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/disk/disk16":
The path of the physical volume refers to a device that does not
exist, or is not configured into the kernel.
vgchange: Couldn't activate volume group "/dev/vg05":
Quorum not present, or some physical volume(s) are missing.

vgchange: Warning: couldn't query physical volume "/dev/disk/disk16":
The volume group is not activated
vgchange: Warning: couldn't query physical volume "/dev/disk/disk19":
The volume group is not activated
vgchange: Warning: couldn't query physical volume "/dev/disk/disk17":
The volume group is not activated
vgchange: Warning: couldn't query the list of physical volumes.
Checking vxfs file systems
vxfs fsck: V-3-20913: /dev/vg00/lvol3:sanity check: root file system OK (mounted read/write)
vxfs fsck: V-3-25742: /dev/vg05/lvol1:sanity check failed: cannot open /dev/vg05/lvol1: No such device or address

for all logical volumes in vg05.

Patch Level is : B.11.31.1003.347a

fcmsutil /dev/fcd2

Vendor ID is = 0x1077
Device ID is = 0x2422
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12D6
PCI Mode = PCI-X 266 MHz
ISP Code version = 5.2.0
ISP Chip version = 3
Topology = PRIVATE_LOOP
Link Speed = 2Gb
Local N_Port_id is = 0x000002
Previous N_Port_id is = 0x000002
Local Loop_id is = 124
N_Port Node World Wide Name = 0x5001438000ad8d41
N_Port Port World Wide Name = 0x5001438000ad8d40
Switch Port World Wide Name = N/A
Switch Node World Wide Name = N/A
N_Port Symbolic Port Name = hjhpux04_fcd2
N_Port Symbolic Node Name = hjhpux04_HP-UX_B.11.31
Driver state = ONLINE
Hardware Path is = 1/0/6/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) fcd B.11.31.1003.01 Mar 3 2010

Now to add to this, If i do ioscan -fnC disk then disks get scaned then vgchange -a y vg05 works and i am able to mount and use the volumes properly, but are not getting mounted after reboot.

Any help is highly appriaciated.

Thanks,
Balaji

[Kapil Jha]

This event says
**********************
Severity: CRITICAL
Description: Fibre Channel Driver received a Fatal Error Notification.
Cause: A possible cause is a Mailbox Command time out.
Action: Please see Preceding Error messages in the log files to isolate the cause. The Fibre Channel Driver will now perform fatal error handling. If there was no unretrieved dump for the adapter port, the driver will have collected a new dump for the adapter port. This dump should be retrieved as soon as possible using the command: /opt/fcms/bin/fcmsutil dump_saved_state > Please save the dump for HP analysis. If auto-recovery is turned on, the driver will automatically re-initialize the adapter port. If auto-recovery is turned off, the adapter port will not automatically be re-initialized. In this situation, please use the following command to re-initialize the adapter port: /opt/fcms/bin/fcmsutil reset
***********************

I hope this would help.

BR,
Kapil+

[Benoy Daniel]

Please verify the FC driver version.

[Balaji Borlepawar]

Thanks for all replies, i have already resetted the card but no luck.

For your information,

Card Model is : HP AB378-60101 & Drivers installed on the server are,
FibrChanl-00 B.11.31.0903 FibreChannel;HW=A6795A,A5158A
FibrChanl-01 B.11.31.1003.01 FibrChnl;HW=A6826A,A9782A,A9784A,AB378A/B,AB379A/B,AB465A,AD193A,AD194A,AD300A
FibrChanl-02 B.11.31.0903 FCLP;HW=AD299A,AD355A,AD221A,AD222A,AD393A,AH402A,AH403A,403621-B21,456972-B21

FibrChanl-01 is the latest installed drivers.

Thanks Balaji

[Balaji Borlepawar]

Hi guys, Good News...!!!

problem got fixed by rebooting HP Storageworks Modular Smart Array 1000 which was connected to this host.

Thanks for all the help.

Balaji