P2000 G3 iSCSI & vSphere ESX multipath

[Fatih Celebi]

The scenario

P2000 G3 iSCSI Dual Controller. 4 iSCSI port on each controller = 8 iSCSI Targets
Between the P2000 and ESX servers we will have 2 physical switches.
Lets call them Psw1 and Psw2
P2000 iSCSI controller A will have 2 connections to Psw1 and 2 to Psw2
P2000 iSCSI controller B will have 2 connections to Psw1 and 2 to Psw2


The ESX servers will have 2 dedicated physical nic for iSCSI traffic.
Lets call them pnic1 and pnic2.
There will be 2 vmkernel ports for iSCSI (VMK_iSCSI_1 and VMK_iSCSI_2 )
VMK_iSCSI_1 will have the pnic1 as the active adapter and pnic2 as unused
VMK_iSCSI_2 will have the pnic2 as the active adapter and pnic1 as unused

pnic1 will be connected to Psw1 physical switch
pnic2 will be connected to Psw2 physical switch

With this design I am trying to create a redundant and load balanced (with multipathing) iSCSI storage network.

I have done this with a Lefthand solution where it works perfect as there is only one iSCSI target. ( a VIP address )
But with P2000 G3 I will have 8 iSCSI targets. Therefore my question is if it is possible to enter all the 8 iSCSI in iSCSI vmhba adapter Dynamic discovery tab. And then there is the question with the Active/Passive role of the controllers ? Is it possible to use round robin with P2000 G3 iSCSI ? Or should I use MRU/Fixed and use separate paths on each ESX server ?

P.S. This thread has been moved from VMware>General to ESX. -HP Forum Moderator

[Florian Schmutz]

Re Up.

So the question is more is it possible to trunk all 4 Ports each controller to a Switch.
If yes is it also possible to give here a VIP or is it only possible to adress each Iscsi Port with a own adress.

The web If shoes ips for each nic but i can give one ip multible interfaces..

Im happy for any Feedback

[Haris Saracevic_1]

I have exactly the same layout, and the same question. It's a shame no one answered!
According to HP document "Configuration best practices for deploying VMware vSphere 4.1 on the HP P2000 G3 MSA Array combo controller" (page 20) Round Robin is best path selection policy.

[J. T. Kirk]

Hello, i've recently added 4 MSA2312 P2000 G3 storages to my VSphere 4.0 infrastructure and i can confirm that any P2000 is seen as a single target presenting 7 devices (2 controllers plus 5x2TB LUNs) with a total of 48 paths ((1 controller + 5 LUNS)x 8 iscsi ports).

When i map the ISCSI target for the first time in the ESXi Server it automatically detects the remaining 7 targets.

In my environment the ESXi has 4 dedicated VMKNics (each one on a different subnet) and corresponding to a different physical switch.

My problem in this environment is that i expected better performances, storage vmotion of vms is really slow.

Best regards Roberto Traversi.

[Antonio Milanese]

Hello,

MSA P2000 ULP IS indeed an T10/SPC-3 ALUA compliant implementation so the real problem here lies in the fact that ESX does not have a builtin SATP rules to map P2000 after inquiry/RTPG

Now for multipath to work correctly first you'll need to define two non routed subnet and map evenly each controller port to each different subnet (and maybe VLAN).

Then you have to manually add SATP rules because default one incorrectly map to VMW_SATP_DEFAULT_AA or VMW_SATP_LOCAL for SAS version.

log on into esxi host (or use vMA) and add a new satp rule:

esxcli nmp satp addrule -satp="VMW_SATP_ALUA" -vendor="HP" -model="P2000" -description="HP P2000 with ALUA support" -claim-option tpgs_on

add a default RR PSP policy:

esxcli nmp satp setdefaultpsp -psp VMW_PSP_RR -satp VMW_SATP_ALUA

check that all it's fine:

esxcli nmp satp list

and then tune RR policy to spread traffic for each presented lun

for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do echo $i ; esxcli nmp roundrobin setconfig -type "iops" -iops=1000 -device $i ;done

it's advisable to mask off enclosure LUN0 or change host profile to "OpenVMS" so it's not presented at all.

Regards,
Antonio

[Fatih Celebi]

Hi,

Thankyou everyone who replied. I forgot to have mail notification on my post so I almost forgot this thread. I found it again while I was "googling" it again.

I did setup the case I described in my first post. All the 8 iSCSI ports are enabled and used and the LUN is presented through all iSCSI ports.
The strange thing is that I got only 5 paths to the LUN from ESXi ? All the iSCSI ports and both vmknic is on same subnet and no vlan used.
From the ESXi I can ping port A1/A2 and B3/B4. These ports are connected to the first switch Psw1 which only one vmknic VMK_ISCSI1 is connected but why does it not try to go out through vmknic2 VMK_ISCSI2 to the A3/A4 and B1/B2 which are connected to the other switch Psw2 ?

All LUN is using round robin.

[Antonio Milanese]

Hello Fatih,

5 (odd!) paths seams strange..have you masked the enclosure lun0 ?

could you post output of:

vicfg-mpath -l
esxcli swiscsi nic list -d
esxcli swiscsi session

Did you use dvSW ?
Since you use a single subnet double check vmknic-to-vmnIC bindings/mapping


Regards,
Antonio

ps: I prefer different subnets (and vlans) for controller/ports even if one single flat subnet it's valid: IMHO it's more predictable on the network (LACP,STP,etc,) and reduce sessions*path*target

[Fatih Celebi]

Hi and thankyou for your reply,

I don't see any option to mask lun 0 from the P2000. I have read somewhere that you could use vms as a profile which masks lun0. But I don't see how that is related to the 5 paths. What I mean is every lun have 5 paths , not total. With 4 luns and lun 0 I got 5 devices and 25 paths total.

I started with 2 different subnets and vlans with 2 iscsi port/ctrl on each vlan but I never got more than 2 paths on every lun so I reverted back to one subnet. Now I at least has 5 paths but it seems strange.

I don't have access to the environment and I can not post the output of the commands right now.

But I remember the output and both the iscsi vmk adapters are bound to iscsi vmhba adapters.

Here is the exact commands i used to configure it through vma.

vicfg-vswitch -a vSwitch2
vicfg-vswitch -m 9000 vSwitch2
vicfg-vswitch -A iSCSI1 vSwitch2
vicfg-vswitch -A iSCSI2 vSwitch2
vicfg-vswitch -L vmnic3 vSwitch2
vicfg-vswitch -L vmnic4 vSwitch2
vicfg-vswitch -p iSCSI1 -N vmnic4 vSwitch2
vicfg-vswitch -p iSCSI2 -N vmnic3 vSwitch2
vicfg-vmknic -a -i 10.0.0.115 -n 255.255.255.0 -m 9000 iSCSI1
vicfg-vmknic -a -i 10.0.0.116 -n 255.255.255.0 -m 9000 iSCSI2
vicfg-iscsi --swiscsi -e
vicfg-scsidevs -a
esxcli --server ub1esxi01 swiscsi nic add -n vmk1 -d vmhba38
esxcli --server ub1esxi01 swiscsi nic add -n vmk2 -d vmhba38
vicfg-iscsi --discovery --add vmhba38 --ip 10.0.0.11
esxcli --server ub1esxi01 nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR
esxcli --server ub1esxi01 corestorage claimrule load
esxcli --server ub1esxi01 corestorage claimrule run


As you can see I add the iSCSI target 10.0.0.11 and after a rescan all 8 iSCSI port ip is available in the iSCSI vmhba adapter. But even though everything is on same subnet one vmknic/vmnic is not available to ping every iscsi ip. The strange part is that it can ping 2 ip's from ctrl-a and 1 ip from ctrl-b.

[Antonio Milanese]

Hello,

yes may be it's totally unrelated but 5 paths are looking too much like half total paths plus enclosure lun =)
and yes you can mask at storage level setting host profile to OpenVMS (lun0 not mapped)..or you can build a claim rule where you mask the lun 0:

http://pubs.vmware.com/vsp40/wwhelp/wwhimpl/js/html/wwhelp.htm#href=fc_san_config/t_mask_paths.html

anyway you have 1 vSW and 2 vmk ports..well I must stress you to double check vmk-vnic binding and vnic teaming override: yes I read yours first post when you've said that teaming (active+unused) is set at vmk port level but check again the output of 3 above esxcli commands

Well the commands are almost ok but i do not see the "satp addrule" part where you add a SATP rule so ESX set the correct rule/policy as ALUA after inquiry/rtgp..may be just a cut&paste typo.

If all above are ok perhaps the problem lies at the network layer (we dont ignore pings=).. and just for curiosity are we talking about blades or "plain" servers?


Regards,
Antonio

[Uwe Zessin]

You have configured the vMotion port on vmk0?

I have seen that in some cases the iSCSI SW initiator implicitly binds to that port, too, which creates additional paths.

# esxcli swiscsi nic list --adapter vmhba38

(Please put in a .TXT file and attach if you want to show us - do not cut&paste in the Reply-box. Thanks!)