LH Starter SAN with LH DSM - only using one of two NICs for iSCSI i/o

[IainS]

Basic problem with our setup is that only one of the two iSCSI NICs is ever being used as a time for iSCSI i/o. I think this might be normal for a single session/application but I'm expecting multiple applications at least to use the other NIC at the same time. This never seems to happen, but I think it should with the nodes set to ALB bonding and using the LH DSM for iSCSI.

If I disable one of the NICs or the port on the switch then it does fail-over to the other NIC (after about 3 seconds and opening 3 more iscsi sessions to the volume) but I can see no evidence of load balancing over more than one NIC at all.

I'm thinking this can't be right. One of the ideas of LH is that you get more peformance if you add more nodes. I thought that the DSM opened more sessions to each node and from each NIC and used multiple of them. This can't be true if you only ever get i/o from a single server over a single NIC can it?

Am I expecting too much?

Our setup is as follows:

HP StorageWorks P4300 G2 7.2TB SAS Starter SAN
2 Nodes each with 8 x 450gb 15K SAS hard drives (RAID5 - Network RAID10 for all volumes)

2 x HP Procurve 2910al-24G Switch

2 x HP DL160 G6 LFF SAS/SATA Server
Intel E5620 Xeon Quad Core 2.4 Ghz Proc
36GB Mem
Embedded HP NC362i PCI-E Dual Gigabit Server Adapter
HP NC375T PCI Express Quad Port Gigabit Server Adapter
2 x 300GB 6G SAS 15K LFF DP Hot Plug Enterprise Drive (mirrored)

Server Networking
2 NICs for iSCSI
- 1 going into each switch
- VLAN only for iSCSI
1 NIC for CSV - in own VLAN
1 NIC for Live Migration - in own VLAN
1 NIC for Parent and Child VMs - in default VLAN
1 NIC spare (have plans for later - probably will also use for Parent Partition mgmt in future)

Each node has 1 IP address set up with ALB bonding (as recommended in docs)
LH iSCSI DSM installed on each server, and rebooted.
iSCSI connected set-up as per documentation with MPIO on and a connection for each NIC pointing to SAN Cluster VIP. More connections are added by the LH DSM although it seems to usually do 3-5 for one node and often only one for the other node (but seems to change sometimes possibly due to activity). Both iSCSI NICs show in the Initiator IP/Port

Originally had been using file copies of large files for load testing but are now using one or more instances of CrystalDiskMark 3 since that seems to effectively work similarly and try different i/o patterns also. Seems to make no real difference to load balancing - which does not seem to be happening anyway.

[IainS]

Found another thread that was from April last year that describes the same thing that others were finding.

There must be a solution to this or else wouldn't everyone here be complaining about this like crazy? Can anyone shed some light on this?

We are going to try doing without the DSM to see if that helps. It sounds like it might, and I think I saw both NICs being used during testing before I installed the Lefthand DSM but not 100% confident of that.

[IainS]

Seems like MS MPIO works rather well. Now get both iSCSI NICs at 99% for a single file operation (nearly 200 MB/sec).

You do have to run the MPIO (separate app in 2008 R2) discover paths option for Multipathing to work or else you get more than one drive showing up for multiple connections which is not good. But after that (forces a reboot) it works well. SAN is using ALB bonding.

[Robert Lennie · Accepted solution]

The current DSM for the P4000 does not support active/active iSCSI connections for MS MPIO. This functionality will be available in the 9.0 release of SAN/iQ. In order to take advantage of both NICs (still only using active/passive for redundancy) you can define different primary paths in the MS initiator settings for different LUNs. Although this is not ideal - it still provides redundancy and makes use of both NICs - until added functionality becomes available.

[Justin Vashisht]

Robert,

Any idea on a rough ETA for 9.0?

[IainS]

Thanks for that response, I have also spoken to HP (I guess it was LeftHand support) and they indicated that it was supposed to work that way. However neither of us was really convinced that it was a good thing and I guess it being in the next release suggests HP/LeftHand think that way too.

The main issue I have is that this is not clear in the documentation and so I was convinced we had screwed something up. Because it seems so wrong it would be nice to explicitly state it is mean to only work in failover.

You can get full bandwidth on both NICs with the MS ISCSI DSM but it does seem to only connect to a single node, so IOPs with smaller random i/o pattens appear to be a bit better with the LH DSM installed.

With our current setup with 2 nodes and 2 hosts I am not sure which one to use. But with more nodes or more hosts I think the LH DSM would be significantly better.

Bring on the new version and hopefully we can get full bandwidth and the best performance too.

[IainS]

Oh and LH support gave an indication of late this year or early next year for v9.

Not sure if it will work out to be the case though.