How is "cX" number determined for disk in SAN?

[Stuart Abramson]

We are moving from a "flat SAN" to a "core-edge SAN" architecture.

The "cX" number of the disks is determined, in part, by the Domain ID, Area ID and Port ID of the switch that connects the server HBA to the Symm FA. However, in our new architecture, there will be TWO switches between the HBA and the Symm FA: The Core Switch connects to the Symm and the Edge switch connects to the server HBA, and the two switches are ISL-d togehter.

So, my question is, which Switch is it that influences the "cX" number? The Core/FA switch, or the Edge/HBA switch?

[Stuart Abramson]

Clarification: One of the reason's that I'm asking this is to help determine if the "cX" number of my existing disks will change when we make this change. The old switch will become the "Core" and the new switch will become the "Edge".

So if the FA-facing switch determines the "cX" number (Domain ID, Area ID, Port ID), I'm fine. But if the HBA-facing switch determines the "cX" number, then I will have to do vgimport/vgexports.

With my luck, it will be divided between both HBA-facing and FA-facing switches...

[Slawomir Gora]

Hi,

I think that your cX will change and
vgexport/vgimport will be necessary.

[Geoff Wild]

THis may help you...

How to decode those device numbers:

e.g 0x1f0d0700

The first 2 hex digits (1fhex - 31decimal) indicate the major device number. Do an lsdev and look for "31". You will find that major block device 31 is SCSI disk. Thus this is a /dev/dsk device node.

The next 2 hex digits (0dhex - 13 decimal) indicate the bus "instance" number or controller number; "c13" in this case.

The next hex digit (0) indicates the SCSI ID or target. "t0" in this case.

The next hex digit (7) indicates the LUN (d7) in this case.

The remaining 2 hex digits are device driver specific.

In summary, 0x1f0d0700 decodes to
/dev/dsk/c13t0d7.

Rgds...Geoff

[Sundar_7]

.

[Sundar_7]

excerpts from my notes. Can help you.

Fibre channel is a link technology that allows other protocol to run on top of it. With respect to disks, SCSI -2 is running over fibre-channel link.This gives best of the both worlds â distances of the LAN and the ease of use of SCSI.

Important thing to remember here is that, we will be using SCSI-2 addressing for the devices. This means the instance number used in the device file of the disk is the instance number of the ext_bus the disk is connected to.

Remember, instance number of the disk itself is not used in the device file.

root:(root)> ioscan -fnkC ext_bus | grep 0/12
ext_bus 4 0/12/0/0.1.30.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 5 0/12/0/0.1.30.0.1 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 11 0/12/0/0.1.30.0.3 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 6 0/12/0/0.1.30.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
root:(root)

[Sundar_7]

fcparray is the interface using the SCSI protocol to communicate with the LUNs configured. The entry described as â Device Interfaceâ is simply a reference for the array itself.

0/12/0/0.1.30.0.0

the format of the hardware path can be broken down in to the following





The hardware path of the fibre channel interace card. Could be seen with ioscan â fnkC fc command.

root@server16:(root)> ioscan -fnkC fc | grep 0/12
fc 0 0/12/0/0 td CLAIMED INTERFACE HP Tachyon TL/TS Fibre Channel Mass Storage Adapter
root@server16:(root)>



Port ID is a 24 bit address broken in three components

1) Domain (switch number) = 1
2) An Area (Physical port on the switch) = 30
3) A Port (always a 0 in a switched fabric SAN)

1.30.0

Note about area: In some older SAN switches, there is a setting known as Core Switch PID Format. When turne off, we would add 16 to the physical port number to give the area (30 â 16 = 14 is the physical port number in the switch the server is connected to). When turned on, we simply use the physical port number on the switch.


If the server is connected to a different port number on the SAN switch, then the system will see different ext_bus entries with different instance numbers and thus the device files of the disk will change. This means lots of vgexport and vgimport is needed to get the volume groups back online again.

SCSI-Address

The last character in the HW Path is known as the Virtual SCSI bus (VSB) address. Fibre channel supports SCSI-2 protocol. The last component of the address can be visualized as the Virtual SCSI bus that can support upto 128

[Sundar_7]

root@server17:(root)> ioscan -fnkC fc
Class I H/W Path Driver S/W State H/W Type Description
=================================================================
fc 0 1/4/0/0 td CLAIMED INTERFACE HP Tachyon TL/TS Fibre Channel Mass Storage Adapter
/dev/td0
fc 1 1/12/0/0 td CLAIMED INTERFACE HP Tachyon TL/TS Fibre Channel Mass Storage Adapter
/dev/td1
root@server17:(root)>

Instance number of the Fibre channel card itself plays no part in the device file name of the disks attached to the FC. Remember, a SCSI disk is attached to an ext_bus interface.


Now let us look at the address of the disk itself.

disk 7 1/4/0/0.1.30.0.0.0.4 sdisk CLAIMED DEVICE EMC SYMMETRIX /dev/dsk/c6t0d4 /dev/rdsk/c6t0d4

Hardware addressing follws SCSI-2 address which means we need to use target and LUN to address a particular disk.

Now here is an example of how to convert a LUN number into a SCSI address.

SCSI address is composed of a target and LUN number

Target = 4 bit address means valid values of 0 to 15
LUN = 3 bit address means valid values of 0 to 7

Let us take an LUN number of 20 for example, here is the simple way to calculate the target and SCSI LUN address

LUN = 20

1) Divide the LUN number by 8. This gives us the SCSI target address

Target = 20 / 8 = 2

Remainder gives us the LUN number = 20 % 8 = 4

SCSI address = 2.4



Complete path of the disk : 1/4/0/0.1.30.0.0.0.4

1/4/0/0 â HBA HW address

1= Domain ID â Switch Number

30 = Physical port number

0 = Always 0 in a switched fabric configuration

0 = Virtual SCSI Bus address

0 = Scsi target

4 = SCSI LUN


The problem we have here is that LUN numbers assigned by the disk array can be more than 128. HP-UX introduced something called Virtual SCSI bus to address this limitation.

VBUS is a 4-digit number. That means we can allocate 16 VSB * 128 LUNS per VSB = 2048 LUNS per fiber channel card.

Now we need to follow a different formula to calculate the SCSI target/LUN number of a given LUN.

1. Virtual SCSI Bus starts at 0
2. If the LUN number is < 128 , skip to step 6
3. If the LUN number is > 128, increase VBUS by 1. Reduce LUN number by 128
4. If the LUN number is still > 128, go back to step 3
5. Divide the LUN number by 8 â this gives the SCSI target number
6. the remainder gives us the LUN number

Say in out SAN, LUN=200, let us work out the 3 components of the SCSI array

1) LUN=200
2) LUN > 128 , So increase VBUS=1 and substract 128 from LUN. LUN=200-128 = 72
3) LUN < 128 , so divide by 8 = 72/8 = 9 â SCSI target number
4) Remainder is 0 â SCSI LUN Number

SCSI-2 address for the array LU

[Stuart Abramson]

I want to thank you all for your help.

Let me just repeat my question:

..."So, my question is, which Switch is it that influences the "cX" number? The Core/FA-facing switch, or the Edge/HBA-facing switch?

[Tim D Fulford]

Hi

Basically every time a disk is added the "instance" number is increased. so a given switch, port ID etc will not determin the controller instance number (cX), but the order in which they are discovered. If you put two devices simultanously the next available number will be assigned to the lowest based upon the H/W path

so 0/0/10/0.3.49.0.0.1 devices will be before 0/0/12/0.1.39.0.0.1 devices

If tou (as in your case) have same devices seen down the same FC adaptor then

0/0/10/0...0..

the lowest switch ID will get the next controlelr instance number.

HTH

Tim

The switch with the lowest