MSA 2040: Discovery shows ports that are not mapped

[Larsen0815]

Hi,

I'm a total newbie to SAN , but have managed to get it working so far. However, some things don't look right to me.

I have attached two hosts via DAC (two cables each). vm03 to controller A1 and B1, vm04 to controller A3 and B3. In the management utility, I have mapped vm03 to port 1, and vm04 to port 3.

For completeness in case I might add other hosts later, I have also configured IP adresses for the other ports on the MSA 2040. I have setup open-iscsi and multipath on vm03 (a Proxmox host). vm04 will follow later.

Now, when I run "iscsiadm -m discovery -t sendtargets -p 10.0.1.100 -I iscsi_enp2s0f0", I would expect to only get the IPs for A1 and B1, but get this:

10.0.1.100:3260,1 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.1.110:3260,2 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.2.120:3260,3 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.2.130:3260,4 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.3.140:3260,5 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.3.150:3260,6 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.4.160:3260,7 iqn.1986-03.com.hp:storage.msa2040.15472745cf
10.0.4.170:3260,8 iqn.1986-03.com.hp:storage.msa2040.15472745cf

Of course, there is no way to connect to the other 6 addresses...

Is this expected behaviour? Might this be the result of some kind of caching?

(I had configured a mapping to all four ports before, but have started over again and deleted everything related I could find)

[support_s]

System recommended content:

1. HPE MSA 1040, MSA 2040, and MSA 2042 Storage GL225R003 Firmware Release Notes

2. HPE MSA 2040 User Guide

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[JonPaul · Accepted solution]

@Larsen0815 
Yes this is compelely expected behavior. During the discovery one of the SCSI inquiries retrieves ALL the host port IP addresses from the MSA.  This is reported to the host whether they are accesible via the network to the host or not.  Or whether there is a volume presented from those ports. It does not cause any problems but you do want to validate that you have multipath showing you have both an optimized and unoptimized PATH.

[Larsen0815]

Ok, good to know.

Multipath is configured and working:

atl-vm03:~# multipath -ll
3600c0ff000277b3d819b3c6201000000 dm-16 HP,MSA 2040 SAN
size=5.2T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
|- 7:0:0:0 sdb 8:16 active ready running
`- 8:0:0:0 sdc 8:32 active ready running

Still wondering though, when I restart open-iscsi, I get these warnings:

May 06 11:23:21 atl-vm03 iscsid[717396]: connect to 10.0.2.120:3260 failed (No route to host)
May 06 11:23:21 atl-vm03 iscsid[717396]: connect to 10.0.2.120:3260 failed (No route to host)
May 06 11:23:21 atl-vm03 iscsid[717396]: connect to 10.0.3.150:3260 failed (No route to host)
May 06 11:23:21 atl-vm03 iscsid[717396]: connect to 10.0.3.150:3260 failed (No route to host)
...

Why is it trying to connect to the other ports?

It is also trying to connect to the ports I use (10.0.1.100 and 10.0.1.110) and fails for 10.0.1.110. Probably because it's not using the right interface as I can ping .100 and .110 (have to use -I for the second one):

atl-vm03:~# ping 10.0.1.100
PING 10.0.1.100 (10.0.1.100) 56(84) bytes of data.
64 bytes from 10.0.1.100: icmp_seq=1 ttl=64 time=0.229 ms

atl-vm03:~# ping 10.0.1.110 -I enp2s0f1
PING 10.0.1.110 (10.0.1.110) from 10.0.1.210 enp2s0f1: 56(84) bytes of data.
64 bytes from 10.0.1.110: icmp_seq=1 ttl=64 time=0.284 ms

Maybe you have an idea even though open-iscsi is not an HP product. Otherwise, I would of course consult the open-isci mailing list.

[Larsen0815]

Nevermind. Just minutes after my previous reply, I got an idea that this was caused by the automatic start of all those targets. Set startup to manual for the unused ports and solved that problem

Just to verify: The ports are also reported to the host even when they are not mapped in the management utility?

[JonPaul]

@Larsen0815 
Yes, the host ports which have an IP address other than 0.0.0.0  will show up in the SCSI VPD pages which are queried during discovery. And will then show up in open iSCSI.
Looking at your multipath output, I'm wondering if the multiple paths are being discovered correctly.  It appears that you have 2 paths  sdb and sdc  but they are both set to a PRIO of 30.  Which would indicate they are both the same ALUA state.  Since they come from independent controllers, 7 and 8,  it looks like you have 2x HBAs connecting to the same MSA host port.  I think that your network issue causing the ping issue may be connected.

[Larsen0815]

Ok, thanks.

Multipath: The issue with open-iscsi arose because I had set all discovered targets to automatic startup. Therefore it was trying to connect to .110 over both interfaces. I disabled all of them but two (.100 on interface 0, and .110 on interface 1) and restarting open-iscsi doesn't show any problems anymore.

Not sure, if that's what you meant regarding the ping issue, though. 

[JonPaul]

@Larsen0815 
When I look at your multipath -ll output I see 2 equal Paths:

|- 7:0:0:0 sdb 8:16 active ready running
`- 8:0:0:0 sdc 8:32 active ready running

Both under the same 'prio' value

prio=30

This would indicate to me that you are ONLY connected to one controller
Here is my multipath output:

mpathe (3600c0ff0001be33453df786201000000) dm-13 HP,MSA 2040 SAN
size=93G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 5:0:0:1 sdk 8:160 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 6:0:0:1 sdl 8:176 active ready running
There are 2 different 'prio' groups which indicate I have a connection to both the Optimized PATH and the un-optimized Path.
One other way to look at the PATHs is to pull the CLI command in XML output:
CLI>  set cli-parameters api pager off
CLI> show initiators
Look at the output for the host port bits:
<PROPERTY name="host-port-bits-a" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr A">8</PROPERTY>
<PROPERTY name="host-port-bits-b" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr B">8</PROPERTY>
The number is a bitmap of the ports 1 == 1, 2==2, 1 and 2 == 3...  in my case port 4 == 8.  You can see my host port bits for the initiator show a login to the A controller port 4 and B controller port 4

[Larsen0815]

This is the redacted output of "show initiators":

<COMP G="0" P="1"/>  <OBJECT basetype="initiator" name="initiator" oid="1" format="rows">
    <PROPERTY name="nickname" type="string" size="255" draw="true" sort="string" display-name="Nickname">atl-vm03</PROPERTY>
    <PROPERTY name="host-port-bits-a" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr A">1</PROPERTY>
    <PROPERTY name="host-port-bits-b" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr B">1</PROPERTY>
  </OBJECT>
<COMP G="0" P="2"/>  <OBJECT basetype="initiator" name="initiator" oid="2" format="rows">
    <PROPERTY name="nickname" type="string" size="255" draw="true" sort="string" display-name="Nickname">atl-vm04</PROPERTY>
    <PROPERTY name="host-port-bits-a" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr A">4</PROPERTY>
    <PROPERTY name="host-port-bits-b" type="uint32" size="8" draw="true" sort="integer" display-name="Host Port Bits Ctlr B">4</PROPERTY>
  </OBJECT>

The multipath output comes from this. Comment out the following in "/etc/multipath.conf"...

devices {
    device {
        vendor "HP"
        product "MSA 2040 SAN"

        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        failback                immediate
        no_path_retry           18
    }
}

...and after a multipathd restart you get this:

3600c0ff000277b3d819b3c6201000000 dm-16 HP,MSA 2040 SAN
size=5.2T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 7:0:0:1 sdb 8:16 active ready running

The above configuration is based on what I have seen on different websites to utilize both controllers.
Should I stick to the default configuration?

[JonPaul]

@Larsen0815 
Likely a mixture of both,  there is one default which is possibly not correct.
**failback**  -  immediate  should be used, I believe that 'manual' is the default  (to see defaults:  multipath -t).  immediate tells the system to immediately failback to the highest priority group when it is available, this would be going back to the optimized paths
path_selector  -  round-robin  or  service-time (default)  are ok
path_grouping_policy  -  this should be group_by_prio.  Setting this to multibus makes all paths equal,  this setting with round-robin will likely lead to some performance degradation as the system will use the non-optimized paths equally to the optimized paths.

FYI - the host-port-bits set to '4' indicates that your host has logged in to port 3 on both controllers.  This shows that the physical connections are correct.

An optimized path is one from the controller which 'owns' the volume, Pool, disk-group.  An unoptimized path has to transfer the I/O request inside the MSA from one controller to the other to be services and then back to the original controller to respond to the host.  It can incur a small to medium performance penalty doing the handoff inside the controllers.

[Larsen0815]

I have changed "path_grouping_policy" to "group_by_prio". I probably should also change "round robin" to "service-time", shouldn't I?

Output from "multipath -ll" after that change and after having pulled the cable to controller A, port 1:

3600c0ff000277b3d819b3c6201000000 dm-16 HP,MSA 2040 SAN
size=5.2T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 8:0:0:1 sdc 8:32 failed faulty running
`-+- policy='round-robin 0' prio=10 status=active
`- 7:0:0:1 sdb 8:16 active ready running

Regarding the numbers 7 and 8, how can I recognize which path it belongs to (without pulling cables)?

[JonPaul]

Multipath output looks good (except for the failed path due to pulled cable)

For you configuration with only a single PATH in each group, the path-selector does not matter. It will use ALL the paths in the highest priority group with the Path-Selector until they are not available, 1 Path is 1 Path no matter how you load balance. Both Path-Selectors are tested.
Since you stated that the port A1 was disconnected, based on the multipath output your volume was 'owned' by controller A. Therefore match up the IP address that goes to Port A1 and that is your HBA.
You can also MAP the volume to only port A1 and see what is found, then MAP to only B1 and see what is found.  Saves you from running to the datacenter.  Mapping to a specific port can only be done in the CLI (I think).

[Larsen0815]

Ok, thank you very much!

I (hopefully) only have two questions left:
- With every disk group, is it only possible to specify one LUN or can multiple LUNs be defined?
- Mail notifications: How can I configure port 26 for delivery to the mail server? (Using Proxmox Mail Gateway)

(Or should I create a separate thread for this?)

[JonPaul]

@Larsen0815 

Short Answer:  Yes, you can create a lot of volumes on the same disk sets

A disk-group is part of a Pool,  there are 1 or many disk-groups in a Pool.  A volume is a portion of Pool.  You can create up to 512 volumes total.
All volumes are Thin Provisioned so you can allocate more capacity in volumes than physically exists.
A Volume turns into a LUN when you map it to a host.

The email parameters allow setting a different port, users, passwords and encyption methods both in the SMU and CLI.  I don't have experience in your particular mail gateway.  If you have problems create a new discussion and hopefully we can find a solution.

[JonPaul]

Now after I replied to that I realized that I was focused on Virtual Storage and not the possibility that the Gen4 is using Linear storage.  
With Linear Storage you can create multiple volumes on the same disk-group but only up to the capacity of the disk-group (Thick Provisioned)
If you don't see Virtual storage as an option, you are likely using the SMUv2.  SMUv3 was used for virtual storage.  You can change this in the system settings or there is a link on the login page to go to the 'other' Web interface.
You need to be on FW GL200 or greater for Virtual Storage.
GL225 is the latest
www.hpe.com/storage/msafirmware

[Larsen0815]

We are indeed using Linear Storage as we don't have a license for Virtual Storage.

Will have a look at the documentation regarding email settings and open a new thread if I don't succeed.

Again, thank you very, very much for your assistance! Have a nice weekend!

Lars