topic Re: configuring new SAN lun on RH3.0 in Operating System - Linux

configuring new SAN lun on RH3.0

Kevin_31 — Fri, 22 Oct 2004 07:46:04 GMT

Hello experts, points for suggestions which help me get this sorted.

I've read a few threads here and elsewhere on the internet about getting fibre card drivers working on linux, and I've got the module "lpfcdd" loaded now per the instructions on the Emulex website where I downloaded the driver:
_________________________________________
[root@noble dev]# lsmod lpfcdd
Module Size Used by Not tainted
loop 12888 0 (autoclean)
lvm-mod 65344 0 (autoclean)
ide-cd 35776 0 (autoclean)
cdrom 34144 0 (autoclean) [ide-cd]
sg 38060 0 (autoclean)
lpfcdd 296200 0
parport_pc 19204 1 (autoclean)
lp 9220 0 (autoclean)
parport 39072 1 (autoclean) [parport_pc lp]
autofs 13780 0 (autoclean) (unused)
nfs 96912 2 (autoclean)
lockd 60656 1 (autoclean) [nfs]
sunrpc 92124 1 (autoclean) [nfs lockd]
e100 59140 1
floppy 59056 0 (autoclean)
microcode 5248 0 (autoclean)
keybdev 2976 0 (unused)
mousedev 5688 1
hid 22404 0 (unused)
input 6208 0 [keybdev mousedev hid]
usb-ohci 23688 0 (unused)
usbcore 83168 1 [hid usb-ohci]
ext3 95784 2
jbd 56856 2 [ext3]
sym53c8xx 68912 4
sd_mod 13744 8
scsi_mod 116904 4 [sg lpfcdd sym53c8xx sd_mod]
[root@noble dev]#

[root@noble dev]# more /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: HP Model: 9.10GB C 80-F612 Rev:
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: HP Model: 9.10GB C 80-F612 Rev:
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 11 Lun: 00
Vendor: SDR Model: GEM318 Rev: 0
Type: Processor ANSI SCSI revision: 02
Host: scsi2 Channel: 00 Id: 03 Lun: 00
Vendor: DataCore Model: SANsymphony Rev:
Type: Unknown ANSI SCSI revision: 04
[root@noble dev]#
____________________________________

I also have my fibre card doing a login to the fabric and I can see my lun:
____________________________________
[root@noble dev]# more /proc/scsi/lpfc/2
Emulex LightPulse LPFC Driver Version: 2.01g
HBA: Emulex LightPulse LP9002 2 Gigabit PCI Fibre Channel Adapter
SerialNum: MS34691385
Firmware Version: 3.90A7 (C2D3.90A7)
Hdw: 2002606d
VendorId: 0xf90010df
Portname: 10:00:00:00:c9:38:57:85 Nodename: 20:00:00:00:c9:38:57:85

Link Up - Ready:
PortID 0x51b00
Fabric
Current speed 2G

lpfc0t00 DID 051200 WWPN blah:blah:blah:blah:blah:blah:blah:WWNN [root@noble dev]#
__________________________________

But I either need to do a mknod, MKDEV, or something elusive before I can fdisk or pvcreate and start using the lun

[root@noble dev]# lspci
00:00.0 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06)
00:00.1 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06)
00:02.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:07.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:0f.0 ISA bridge: ServerWorks OSB4 South Bridge (rev 50)
00:0f.1 IDE interface: ServerWorks OSB4 IDE Controller
00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller (rev 04)
01:02.0 Fibre Channel: Emulex Corporation LP9000 Fibre Channel Host Adapter (rev
01)
01:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI A
dapter (rev 01)
01:05.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI A
dapter (rev 01)

[root@noble dev]# cdrecord -scanbus
Cdrecord 2.0 (i686-pc-linux-gnu) Copyright (C) 1995-2002 Jörg Schilling
Linux sg driver version: 3.1.25
Using libscg version 'schily-0.7'
cdrecord: Warning: using inofficial libscg transport code version (schily - Red
Hat-scsi-linux-sg.c-1.75-RH '@(#)scsi-linux-sg.c 1.75 02/10/21 Copyright
1997 J. Schilling').
scsibus0:
0,0,0 0) 'HP ' '9.10GB C 80-F612' ' ' Disk
0,1,0 1) 'HP ' '9.10GB C 80-F612' ' ' Disk
0,2,0 2) *
0,3,0 3) *
0,4,0 4) *
0,5,0 5) *
0,6,0 6) *
0,7,0 7) *
0,11,0 11) 'SDR ' 'GEM318 ' '0 ' Processor
scsibus2:
2,0,0 200) *
2,1,0 201) *
2,2,0 202) *
2,3,0 203) 'DataCore' 'SANsymphony' ' ' not present unknown/no device
2,4,0 204) *
2,5,0 205) *
2,6,0 206) *
2,7,0 207) *
[root@noble/]#

DataCore and SANsymphony are correct and refer to my elusive lun, but the fact that it says, "not present unknown/no device" is where I think my problem lies.

fdisk /dev/sdc
[root@noble dev]# lspci
00:00.0 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06)
00:00.1 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06)
00:02.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:07.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:0f.0 ISA bridge: ServerWorks OSB4 South Bridge (rev 50)
00:0f.1 IDE interface: ServerWorks OSB4 IDE Controller
00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller (rev 04)
01:02.0 Fibre Channel: Emulex Corporation LP9000 Fibre Channel Host Adapter (rev
01)
01:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI A
dapter (rev 01)
01:05.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI A
dapter (rev 01)
[root@noble dev]#
_________________________________________

I've installed lshw and it shows:

[root@noble lshw-A.01.08]# lshw -short
H/W path Device Class Description
================================================
system HP NetServer
/0 bus
/0/0 memory BIOS
/0/1 memory BIOS
/0/2 memory BIOS
/0/7 processor Pentium III (Coppermine)
/0/7/e memory L1 cache
/0/7/f memory L2 cache
/0/8 processor Pentium III (Coppermine)
/0/8/0 memory L1 cache
/0/8/1 memory L2 cache
/0/26 memory System Memory
/0/26/0 memory DIMM DRAM Synchronous 133 MHz (7.5 ns)
/0/26/1 memory DIMM DRAM Synchronous 133 MHz (7.5 ns)
/0/26/2 memory DIMM DRAM Synchronous 133 MHz (7.5 ns)
/0/26/3 memory DIMM DRAM Synchronous 133 MHz (7.5 ns)
/0/100 bridge CNB20LE Host Bridge
/0/100/2 eth0 network 82557/8/9 [Ethernet Pro 100]
/0/100/8 eth1 network 82557/8/9 [Ethernet Pro 100]
/0/100/7 display Rage XL
/0/100/f bridge OSB4 South Bridge
/0/100/f.1 storage OSB4 IDE Controller
/0/100/f.1/0 ide0 bus IDE Channel 0
/0/100/f.1/0/0 /dev/hda disk CD-224E
/0/100/f.2 bus OSB4/CSB5 OHCI USB Controller
/0/101 bridge CNB20LE Host Bridge
/0/3 scsi2 bus LP9000 Fibre Channel Host Adapter
/0/3/0 bus SCSI Channel 0
/0/3/0/3 generic SANsymphony
/0/5 scsi0 storage 53c1010 Ultra3 SCSI Adapter
/0/5/0 bus SCSI Channel 0
/0/5/0/0 /dev/sda disk 9.10GB C 80-F612
/0/5/0/1 /dev/sdb disk 9.10GB C 80-F612
/0/5/0/b processor GEM318
/0/5.1 storage 53c1010 Ultra3 SCSI Adapter
/1 scsi1 storage SCSI storage controller
[root@noble lshw-A.01.08]#

If you're still reading this, GREAT! I've also got points for folks with suggestions/opinions about advantages/disadvantages, lessons learned, etc. regarding loading the FibreCard driver as a module vs. buiding it in the kernel. Emulex docs say module is preferred... why? I think I'd rather not customize my kernel any more often than needed, but I'm new to doing H/W stuff of this depth on Linux, so I might be very wrong.

cheers!
Kevin

Re: configuring new SAN lun on RH3.0

Kevin_31 — Fri, 22 Oct 2004 07:59:29 GMT

hwls (full) shows:

*-fiber
description: Fibre Channel
product: LP9000 Fibre Channel Host Adapter
vendor: Emulex Corporation
physical id: 3
bus info: scsi@2
logical name: scsi2
version: 01
clock: 66MHz
capabilities: bus_master
configuration: driver=lpfc
resources: iomemory:fd001000-fd001fff iomemory:fd000000-fd0000ff irq:16
*-channel
description: SCSI Channel 0
physical id: 0
bus info: scsi@2.0
*-generic UNCLAIMED
description: SCSI
product: SANsymphony
vendor: DataCore
physical id: 3
bus info: scsi@2.0:3.0
configuration: ansiversion=4
*-scsi:0
description: SCSI storage controller
product: 53c1010 Ultra3 SCSI Adapter
vendor: LSI Logic / Symbios Logic
physical id: 5
bus info: scsi@0
logical name: scsi0
version: 01
clock: 33MHz
capabilities: scsi bus_master cap_list
configuration: driver=sym53c8xx
resources: ioport:2400-24ff iomemory:fd000400-fd0007ff iomemory:fd002000-fd003fff irq:24
*-channel
description: SCSI Channel 0
physical id: 0
bus info: scsi@0.0
*-disk:0
description: SCSI Disk
product: 9.10GB C 80-F612
vendor: HP
physical id: 0
bus info: scsi@0.0:0.0
logical name: /dev/sda
size: 8678MB
capacity: 10GB
capabilities: 10000rpm
configuration: ansiversion=2
*-disk:1
description: SCSI Disk
product: 9.10GB C 80-F612
vendor: HP
physical id: 1
bus info: scsi@0.0:1.0
logical name: /dev/sdb
size: 8678MB
capacity: 10GB
capabilities: 10000rpm
configuration: ansiversion=2

(amongst other things)

if UNCLAIMED in linux/hwls is like it is in hpux, then I need something like "insf" of "mknod", etc.

cheers!
Kevin

Re: configuring new SAN lun on RH3.0

xyko_1 — Fri, 22 Oct 2004 11:35:11 GMT

Hi Kevin,

first of all lets talk about modules. Modules are dynamic so you may unload and load them when necessary and possible. To unload a module you can't be using it, i.e, you can't have any device attached/using this module. With lsmod you may see if the module is in use or not (count field = 0 => not in use). It's good because if you define/install another lun attached to a inuse card you have to "probe" the scsi channel again, so, if you are using modules (driver not staticaly included in the kernel) you don't have to reboot the server, you just have to remove (modprobe -r) and load (modprobe) the desired module. I hope that my talk helps you, if not please ask more.

Now the Emulex card/driver and the DataCore attached to it.

I don't think your problem is about not having the necessary device (/dev/xxx). You may have problems regarding multi_lun support on kernel or "persistent biding" support on lpfcdd driver.

What is DataCore ? Here in Brasil I have never heard about this. It's a storage device attached to a SAN just like an EMC Clarion ?
Please, send me (attach) some documents about that device. And please, if possible, attach also a complete output from dmesg.
I'll try to help you with my experience with Emulex and EMC SAN storage.

Regards,
Xyko

Re: configuring new SAN lun on RH3.0

Uwe Zessin — Sun, 24 Oct 2004 01:08:19 GMT

Xyko,
DataCore's SANsymphony is an in-band (meaning that the box is in the data path between the servers and the sorage arrays) virtualization solution based on one or more Windows boxes.

Kevin,
I don't know exactly how SANsymphony presents the LUNs. It is possible that LUN 0 is a special non-disk device for SCSI-3 REPORT LUNs and / or in-band communication with the box.

Are you sure you have presented a (virtual) disk to your server? Does SANsymphony have a means to tell it what operating system is running on the server?

Re: configuring new SAN lun on RH3.0

Kevin_31 — Mon, 25 Oct 2004 06:12:41 GMT

Xyko, thanks for the info on modules. Makes sense, what you're saying about modules, but ultimately I'd be hoping to "scan" and "update" the luns on an existing module/fibre card, hopefully without having to interrupt service to the existing lun(s). To unload the module would mean removing access to the existing data on the already configured luns, or am I assuming wrongly. I see it as the kernel, then the module, then the fibre card(s) and lun(s) as the layers that all depend on those above. Or do the luns and linux modules not work this way?

I've read the docs on the Emulex driver from their website and it does support persistent binding (but we aren't using it here). I've just got the driver configured with defaults so far.

I'll lookup "multi_lun support and post back here.

As Uwe says, DataCore SAN Symphony is a product we use to manage our luns. When my SAN administrator is back tomorrow, I'll pose your questions to him, but I'm pretty sure it serves out the luns, but doesn't keep track of specific information about servers which are using these luns (it "pushes" information out, doesn't "pull" much back).

Uwe, I'll look into the lun zero stuff (sounds familiar, but this lun is served as "lun28" target 3, dev4--evidently.

I'll wait to assign points until I find out more (I'd hate to not give enough!).

cheers,
Kevin

p.s dmesg attached

Re: configuring new SAN lun on RH3.0

Kevin_31 — Mon, 25 Oct 2004 06:16:02 GMT

attached dmesg (trying again)

Re: configuring new SAN lun on RH3.0

xyko_1 — Mon, 25 Oct 2004 15:11:13 GMT

Kevin,

I think you understood correctly all about static kernel and dynamic modules. Unfortunately I don't know any way to probe new devices behind a module but recicling it (putting it down and up). If someone knows, please tell me how.

Now the only thing I can tell you is that if your storage administrator says you that Synphony creates more then one LUN (lun 0) you will have to compile your kernel with the option "Probe all LUNs on each SCSI device" under "SCSI Support" on menuconfig.

Regards,
Xyko

Re: configuring new SAN lun on RH3.0

Kevin_31 — Tue, 26 Oct 2004 03:21:50 GMT

Xyko & Uwe thanks loads. You were right about the lun 0 bit. We've reassigned the same lun with lun 0 and chazam:

/0/3 scsi2 bus LP9000 Fibre Channel Host Adapter
/0/3/0 bus SCSI Channel 0
/0/3/0/1 /dev/sdc disk SANsymphony

I'll leave this thread open for a bit in case anyone can suggest the RedHat way of doing the equivalent of HPUX's "insf" command. At a conference last month I was assured that it couldn't be done in RH2.1 but the command is there somewhere in RH3.0. I'll log a call with RH and post the answer here soon.

cheers again!
Kevin

Re: configuring new SAN lun on RH3.0

Uwe Zessin — Tue, 26 Oct 2004 03:27:48 GMT

I am not a Linux expert, but isn't there a mechanism like:

# echo "scsi add-single-device 0 0 4 0" > /proc/scsi/scsi

Re: configuring new SAN lun on RH3.0

xyko_1 — Tue, 26 Oct 2004 07:29:27 GMT

Good news Kevin.
I'm very happy that I could help you anyway.
Regards,
Xyko

Re: configuring new SAN lun on RH3.0

Kevin_31 — Wed, 27 Oct 2004 03:40:23 GMT

aha! I've found the solution many times on various websites, but didn't quite understand how to translate the parameters into my scenario. The solution came, funnily, from a posting on Seagates website about one of their tape drives & Linux.

Here's the elusive bit:
_________________________________________
By default, the Linux Kernel DOES NOT scan past target X Lun 0. This results in
the system not seeing a device at target X Lun 1. Therefore, to correct this you
must add the devices manually.

From a terminal window, run the following command:

"cat /proc/scsi/scsi"

If the library is seen at, Host: scsi0 Channel:00 Id:00 Lun:00, the drives can t
hen be added by adding the following four lines:

echo "scsi add-single-device 0 0 0 1" > /proc/scsi/scsi
echo "scsi add-single-device 0 0 0 2" > /proc/scsi/scsi
echo "scsi add-single-device 0 0 0 3" > /proc/scsi/scsi
echo "scsi add-single-device 0 0 0 4" > /proc/scsi/scsi

Syntax:

echo "scsi add-single-device a b c d" > /proc/scsi/scsi

where:

a == hostadapter id (first one being 0)
b == SCSI channel on hostadapter (first one being 0)
c == ID
d == LUN (first one being 0)

You may then perform "cat /proc/scsi/scsi" and the devices should be seen by the system.
_______________________________________

echo "scsi add-single-device 2 0 0 1" > /proc/scsi/scsi

more /proc/scsi/scsi
Vendor: DataCore Model: SANsymphony Rev: DCS Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 01 Lun: 00

Vendor: DataCore Model: SANsymphony Rev: DCS Type: Direct Access ANSI SCSI revision: 04

Re: configuring new SAN lun on RH3.0

Kevin_31 — Wed, 27 Oct 2004 03:43:57 GMT

my final reply shows how to translate the parameters passed to the /proc/scsi/scsi file via the "scsi add-single-device" command:

If the fC is seen at, Host: scsi0 Channel:00 Id:00 Lun:00, the drives can then be added by adding the line:

echo "scsi add-single-device 0 0 0 1" > /proc/scsi/scsi
Syntax:

echo "scsi add-single-device a b c d" > /proc/scsi/scsi

where:

a == hostadapter id (first one being 0)
b == SCSI channel on hostadapter (first one being 0)
c == ID
d == LUN (first one being 0)