Multipath Connection to EMC SAN faulty

[zissup]

Hello -

I've looked through the threads and got some hints on what to look for and/or try when troubleshooting the issue I'm seeing, but have not resolved the issue.

When I run multipath -ll I see [active][faulty] on two of my paths. This is better than before when they were [failed][faulty]. I've run:
multipath -F
and
echo "1" > /sys/block/sdae/device/delete
then rebooted, which changed the pathes from failed to active, but still faulty.

Here is more information:
sde: checker msg is "readsector0 checker reports path is down"
sdf: checker msg is "readsector0 checker reports path is down"
mpsan1 (360060160a4022100228db9ed8858df11) dm-0 DGC,RAID 5
[size=1000G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=1][active]
\_ 6:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 7:0:0:0 sdd 8:48 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 8:0:0:0 sdf 8:80 [active][faulty]
\_ round-robin 0 [prio=0][enabled]
\_ 9:0:0:0 sde 8:64 [active][faulty]

multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:0:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:0 sdb 8:16 1 [undef] [ready] [orphan]
6:0:0:0 sdc 8:32 1 [active][ready] XXXXXXXX.. 17/20
7:0:0:0 sdd 8:48 1 [active][ready] XXXXXXXX.. 17/20
8:0:0:0 sdf 8:80 0 [active][faulty] XXXXXX.... 3/5
9:0:0:0 sde 8:64 0 [active][faulty] XXXXXX.... 3/5
multipathd>

[root@mach io]# mii-tool -v | grep status
basic status: autonegotiation complete, link ok
basic status: autonegotiation complete, link ok
basic status: autonegotiation complete, link ok
basic status: autonegotiation complete, link ok

We have an EMC CX3-10c.

T.I.A
zissup

[AnthonySN]

What is your server model and OS version (looks like RHEL)
Can you recheck your zoning config on ur switches.

go through the below link and follow the steps.
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/index.html

[Jayakrishnan G Naik]

Hi Zissup,

Is this an HP proliant Server? Which FC card you have, Can you give us the output of the following commands?
#lssd -wl
# lspci |grep -i "Hba card related string"
If you have a qlogic card,
# adapter_info


For me this looks like it can be an issue anywhere from one of your HBA, the cable, SAN switch port , or the zoning configuration. Please ensure the HBA cards on the server are working fine.

Thanks & regards
Jayakrishnan G Naik

[zissup]

This is a Dell R510 server (so, no, not HP) running Oracle VM Server 2.2.0 (a RedHat variant). I do not have some of the commands available that you mention.

lssd -> not available

lspci
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

adapter_info -> N/A

zissup

[Jayakrishnan G Naik]

Hi Zissup,

You may have specific administration commands for the HBA card available on your Dell server, you should install the necessary add-on packages to administer the HBA on your system, or if available you should try to find what is
it.

In the mean time you can try to find whether any issues in other parts, like zoning & issues if any on storage end, cable, san switch ports etc.

Regards
Jayakrishnan G Naik

[Jayakrishnan G Naik]

Hi

Can you tell me which version of linux do you have?

I heard about a tool on redhat which is called lsscsi and you can download the rpm and install it to your system if not available.

lsscsi may help you to administer your connections to SAN.

Regards
Jayakrishnan G naik

[Alzhy]

Do you have the recommended /etc/multipath.conf from EMC? It should be described and you should follow the document "Native Multipath Failover based on DM-MPIO for v2.6.x Linux Kernel and
EMC Storage Arrays Configuration Guide"

[zissup]

Thank you for the replies. My version of linux is Oracle VM Server 2.2.0 which is based off of RedHat (though I don't know which version of RH).

Also, I have the iscsi-initiator-utils rpm installed which gives me access to iscsiadm commands. For example I can list the initiators:
[root@mach yum.repos.d]# iscsiadm -m node
172.16.19.6:3260,2 iqn.1992-04.com.emc:cx.apm00081900361.a1
172.16.19.5:3260,1 iqn.1992-04.com.emc:cx.apm00081900361.a0
172.16.19.8:3260,4 iqn.1992-04.com.emc:cx.apm00081900361.b1
172.16.19.7:3260,3 iqn.1992-04.com.emc:cx.apm00081900361.b0
[root@ovm-wdm38 yum.repos.d]#

I can verify the SAN switch and NICs are working by being able to ping the SAN (thus through the switch); I've also checked the cables connections. I am not familiar with "zoning;" can you elaborate on this?

Thanks,
zissup

[Jayakrishnan G Naik]

Hi Zissup,

The basic definition for zoning is given below.

The process of allocating resources in a SAN to load balance the devices connected to the network. Zoning allows the network administrator to separate the SAN into units and allocate storage to those units based on need. Zoning protects the SAN system from such threats as viruses, data corruption and malicious hackers as the devices in their respective zones are not able to communicate outside the zone through their ports unless given permission. Zoning is done in SAN Switch level.

You need to dig more on net for detailed information. For now, the zoning issue may be causing the paths failure status and SAN admin will be good enough to identify and fix it for you.

Please assign appropriate points for the helpful replies above. Hope you are a new member here. Encourage valuable replies by assigning appropriate points.

Regards
Jayakrishnan G Naik

[Alzhy]

Zissup:

Zoning is simply "allowing" your server's Fibre Channel HBAs to see SAN targets *disks or tapes or fellow HBAs* in a "zone". The objects that are "zoned" are distinguisable objects like port numbers or world wide IDs *WWIDs* of these objects.

For example - your SAN Administrator (since you obviously are not the SAN admin as well) could have a zone like:

mach_cx3_zoneA: mach_hba1 + cx3_ctrlA_port1
mach_cx3_zoneB: mach_hba2 + cx3_ctrlB_port1

On your CX3's management utility, you will likely have access controls set up so your linux server named mach's HBAs aer defined in some sort of a hotgroup where the CX3's arrays virtual disks or luns are presented. Talk to your SAN Administrator for details.

As far as your current predicament - please go over that EMC document and ensure your zoning is indeed correct and your multipath configuration follows what is stipulated by EMC. Every vendor has its own suggested settings -- like HP has the HPDM package for all its StorageWorks line of SAN Arrays.