topic Re: ESXi HBA Queue Depth for a 12 node P4500G2 in StoreVirtual Storage

ESXi HBA Queue Depth for a 12 node P4500G2

pirx4711 — Wed, 14 Mar 2012 08:06:42 GMT

Hi,

what would be the right Queue Depth for a 12 node P4500 cluster in network RAID10? I found this this old post in the forum:

http://h30499.www3.hp.com/t5/HP-LeftHand-P4000/latency-on-P4500-28-8/td-p/4816343

----

Outstanding IOs is Queue Depth Which should be 1.5 to 2 times the number of Disks (SAS Disks) on the Node multiplied by the Number of Nodes in the Cluster.Then you are providing optimum work to your SAN.

----

This would be

2 x 12 (disks) x 12 (nodes) = 288

1,5 x 12 (disks) x 12 (nodes) = 216

...which seems a bit high. Most oft the examples I found are only setting it to 64.

Re: ESXi HBA Queue Depth for a 12 node P4500G2

5y53ng — Wed, 14 Mar 2012 13:51:13 GMT

How many ESXi hosts?

Re: ESXi HBA Queue Depth for a 12 node P4500G2

pirx4711 — Wed, 14 Mar 2012 14:11:26 GMT

6 ESXi 4.1 Hosts each with 2 x 10 GbE Qlogic NICs (NC523SFP).

BTW... the Qlogic NICs show frequent disconnects and firmware hangs even with latest firmware. They will be replaced by Emulex cards soon.

Re: ESXi HBA Queue Depth for a 12 node P4500G2

pirx4711 — Fri, 16 Mar 2012 11:45:09 GMT

I just realized that the 10 GbE NICs we use don't have an option for setting the queue depth at all....

# vmkload_mod -s qlcnic
vmkload_mod module information
input file: /usr/lib/vmware/vmkmod/qlcnic.o
Version: Version 4.0.727, Build: 00000, Interface: 9.0, Built on: Jun 1 2011
Required name-spaces:
Parameters:
skb_heap_max: int
    Maximum attainable private socket buffer heap size for the driver.
skb_heap_initial: int
    Initial private socket buffer heap size allocated for the driver.
heap_max: int
    Maximum attainable heap size for the driver.
heap_initial: int
    Initial heap size allocated for the driver.
jumbo_desc_10g: int
    Jumbo Receive Descriptors for 10G(Should be power of 2). Default:128, Min:32, Max:1024
jumbo_desc_1g: int
    Jumbo Receive Descriptors for 1G(Should be power of 2). Default:64, Min:32, Max:512
rdesc_10g: int
    Receive Descriptors for 10G(Should be power of 2). Default:512, Min:64, Max:8192
rdesc_1g: int
    Receive Descriptors for 1G(Should be power of 2). Default:512, Min:64, Max:4096
tx_desc: int
    Transmit Descriptors in Host(Should be power of 2). Default:1024, Min:64, Max:2048
auto_fw_reset: bool
    Enable/disable auto firmware recovery support. Default:1, Enable=1 ,Disable=0
hw_vlan: bool
    Enable/disable HW VLAN support. Default:1, Enable=1 ,Disable=0
enable_tso: bool
    Enable/disable TSO support. Default:1, Enable=1 ,Disable=0
user_rx_queues: int
    Number of RX queues excluding default Rx Queue, Min:0, Max:7, Current: Check /proc/net/qlcnic/vmnicX/num_rx_queues
multi_ctx: bool
    Enable/disable Rx NetQueues support Default:1, Enable=1 ,Disable=0
lro_enable: bool
    Enable/disable HW LRO Default:0, Enable=1 ,Disable=0
use_msi: bool
    Enable/disable msi interrupts Default:1, Enable=1 ,Disable=0
use_msi_x: bool
    Enable/disable msix interrupts Default:1, Enable=1 ,Disable=0

Re: ESXi HBA Queue Depth for a 12 node P4500G2

5y53ng — Fri, 16 Mar 2012 12:25:24 GMT

Is this the ESX(i) software iSCSI initiator? If so, there are advanced settings on the host for tinkering with the que depth. I only have ESX 3.5 and ESXi 5 hosts at the moment, but go to the configuration tab > advanced settings and you should see some settings in there for the que depth.

Re: ESXi HBA Queue Depth for a 12 node P4500G2

pirx4711 — Fri, 16 Mar 2012 15:17:24 GMT

I did not find the parameter, but I' m also thinking about activating Storage IO Control (SIOC).