How To Associate SCSI Disk Devices with Fibre Channel HBAs

[Paul Gerke]

I am new to Linux. I am trying to connect 5 LUNs on my XP 10000 SAN to a new Linux server (2.6.32-100.26.2.el5). But I can only see the LUNs on one path, but I can't tell which fabric is working and which one I have a problem with. I can see the WWNs for the two HBAs and match them to they switch:
$ cat /sys/class/scsi_host/host3/device/fc_host:host3/port_name
0x21000024ff26faec
$ cat /sys/class/scsi_host/host4/device/fc_host:host4/port_name
0x21000024ff26faed


I can see 5 device files, sdb through sde, that have the proper LUN sizes:
$ cat /proc/partitions
major minor #blocks name

8 0 428212224 sda
8 1 40131 sda1
8 2 10240000 sda2
8 3 104422 sda3
8 4 1 sda4
8 5 417818488 sda5
8 16 155488320 sdb
8 32 155488320 sdc
8 48 155488320 sdd
8 64 155488320 sde
8 80 155488320 sdf
253 0 316768256 dm-0
253 1 101023744 dm-1

And when I run:
$ cat /proc/scsi/scsi*

I can see the appropriate LUN numbers and Vendor/Model Information:
Attached devices:
...
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: HP Model: OPEN-V Rev: 5009
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 00 Lun: 01
Vendor: HP Model: OPEN-V Rev: 5009
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 00 Lun: 02
Vendor: HP Model: OPEN-V Rev: 5009
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 00 Lun: 03
Vendor: HP Model: OPEN-V Rev: 5009
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 00 Lun: 04
Vendor: HP Model: OPEN-V Rev: 5009
Type: Direct-Access ANSI SCSI revision: 03

The multipath software is setup for these 5 device files, but I need the other 5 in order to setup redundant path.

If anybody can point me in the right direction, I would appreciate it.

Paul

[Paul Gerke]

I think I found it:
ls -l /sys/block/*/device
lrwxrwxrwx 1 root root 0 Jun 1 03:43 /sys/block/sda/device -> ../../devices/pci0000:00/0000:00:04.0/0000:03:00.0/host0/target0:2:0/0:2:0:0
lrwxrwxrwx 1 root root 0 Jun 1 10:43 /sys/block/sdb/device -> ../../devices/pci0000:00/0000:00:06.0/0000:05:00.0/host3/rport-3:0-0/target3:0:0/3:0:0:0
lrwxrwxrwx 1 root root 0 Jun 1 10:43 /sys/block/sdc/device -> ../../devices/pci0000:00/0000:00:06.0/0000:05:00.0/host3/rport-3:0-0/target3:0:0/3:0:0:1
lrwxrwxrwx 1 root root 0 Jun 1 10:43 /sys/block/sdd/device -> ../../devices/pci0000:00/0000:00:06.0/0000:05:00.0/host3/rport-3:0-0/target3:0:0/3:0:0:2
lrwxrwxrwx 1 root root 0 Jun 1 10:43 /sys/block/sde/device -> ../../devices/pci0000:00/0000:00:06.0/0000:05:00.0/host3/rport-3:0-0/target3:0:0/3:0:0:3
lrwxrwxrwx 1 root root 0 Jun 1 10:43 /sys/block/sdf/device -> ../../devices/pci0000:00/0000:00:06.0/0000:05:00.0/host3/rport-3:0-0/target3:0:0/3:0:0:4
lrwxrwxrwx 1 root root 0 Jun 1 03:43 /sys/block/sr0/device -> ../../devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0

[Matti_Kurkela · Accepted solution]

Yeah, that's one possible way to do it... but certainly not the most convenient one.

By the kernel version you're running, I guess your Linux distribution is fairly recent, so it might have the lsscsi command available. If that's true, make sure the "lsscsi" package is installed (e.g. "yum install lsscsi") and use the lsscsi command.

In the lsscsi output, there is a [bracketed] set of four numbers in the first column: these match the host/channel/id/lun numbers in /proc/scsi/scsi. The host number also matches the "host*" directories in /sys/class/scsi_host/host*.

According to your "cat /proc/scsi/scsi" output, all your current disks are visible throught SCSI host 3, i.e. the HBA port WWN 0x21000024ff26faec. The other port is not seeing any disks at all at the moment.

If you have verified that the other HBA port has link, you could reset and re-scan it to see if it helps:

Sending a LIP (essentially a FibreChannel HBA reset) to the card with the SCSI host number 4 can be done like this:

echo 1 > /sys/class/scsi_host/host4/device/fc_host:host4/issue_lip

or (equivalent but easier to type):

echo 1 > /sys/class/fc_host/host4/issue_lip

Sending a LIP causes the HBA to reset itself, usually causing the FC link to go down and then back up. You'll see the link down/link up messages in the kernel message buffer (use the "dmesg" command).

Note: Obviously, sending a LIP to the HBA that currently has disk devices in use is a bad idea, if your multipathing is not yet properly configured. Likewise, sending a LIP to both your HBAs in quick succession will cause a connection to your disks to be interrupted even if your multipathing is configured, so be careful with this.

After resetting the HBA, you can tell the SCSI core to re-scan it:

echo "- - -" > /sys/class/scsi_host/host4/scan

Triggering a scan at the SCSI layer should always be safe: the scan never removes or renames any old devices, only adds new devices if any have become available.

The Online Storage Reconfiguration Guide by RedHat should be useful to you:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/index.html

In particular, see Chapter 9: "Scanning Storage Interconnects".

MK

[Matti_Kurkela]

Oops, I forgot:

If the above does not make the alternate paths to appear, it's time to ask your SAN administrator to double-check the SAN zoning/masking settings as related to the "host4" HBA.

MK

[Paul Gerke]

Thanks MK,
I had zoned the wrong XP port, once I zoned the correct one I was able to see the LUNs on both HBAs and multipathing is now setup and functional.

I will take a look at lssci. It looks promising.

Thanks again,
Paul

[Paul Gerke]

Thanks MK,
The lsscsi with its various options does everything I had hoped for to see the connections being made and determine what is missing.

Paul