ESXi HBA Queue Depth for a 12 node P4500G2

[pirx4711]

Hi,

what would be the right Queue Depth for a 12 node P4500 cluster in network RAID10? I found this this old post in the forum:

http://h30499.www3.hp.com/t5/HP-LeftHand-P4000/latency-on-P4500-28-8/td-p/4816343

----

Outstanding IOs is Queue Depth Which should be 1.5 to 2 times the number of Disks (SAS Disks) on the Node multiplied by the Number of Nodes in the Cluster.Then you are providing optimum work to your SAN.

----

This would be

2 x 12 (disks) x 12 (nodes) = 288

or

1,5 x 12 (disks) x 12 (nodes) = 216

...which seems a bit high. Most oft the examples I found are only setting it to 64.

[5y53ng]

How many ESXi hosts?

[pirx4711]

6 ESXi 4.1 Hosts each with 2 x 10 GbE Qlogic NICs (NC523SFP).

BTW... the Qlogic NICs show frequent disconnects and firmware hangs even with latest firmware. They will be replaced by Emulex cards soon.

[pirx4711]

I just realized that the 10 GbE NICs we use don't have an option for setting the queue depth at all....

# vmkload_mod -s qlcnic
vmkload_mod module information
 input file: /usr/lib/vmware/vmkmod/qlcnic.o
 Version: Version 4.0.727, Build: 00000, Interface: 9.0, Built on: Jun  1 2011
 Required name-spaces:
 Parameters:
  skb_heap_max: int
    Maximum attainable private socket buffer heap size for the driver.
  skb_heap_initial: int
    Initial private socket buffer heap size allocated for the driver.
  heap_max: int
    Maximum attainable heap size for the driver.
  heap_initial: int
    Initial heap size allocated for the driver.
  jumbo_desc_10g: int
    Jumbo Receive Descriptors for 10G(Should be power of 2). Default:128, Min:32, Max:1024
  jumbo_desc_1g: int
    Jumbo Receive Descriptors for 1G(Should be power of 2). Default:64, Min:32, Max:512
  rdesc_10g: int
    Receive Descriptors for 10G(Should be power of 2). Default:512, Min:64, Max:8192
  rdesc_1g: int
    Receive Descriptors for 1G(Should be power of 2). Default:512, Min:64, Max:4096
  tx_desc: int
    Transmit Descriptors in Host(Should be power of 2). Default:1024, Min:64, Max:2048
  auto_fw_reset: bool
    Enable/disable auto firmware recovery support. Default:1, Enable=1 ,Disable=0
  hw_vlan: bool
    Enable/disable HW VLAN support. Default:1, Enable=1 ,Disable=0
  enable_tso: bool
    Enable/disable TSO support. Default:1, Enable=1 ,Disable=0
  user_rx_queues: int
    Number of RX queues excluding default Rx Queue, Min:0, Max:7, Current: Check /proc/net/qlcnic/vmnicX/num_rx_queues
  multi_ctx: bool
    Enable/disable Rx NetQueues support Default:1, Enable=1 ,Disable=0
  lro_enable: bool
    Enable/disable HW LRO Default:0, Enable=1 ,Disable=0
  use_msi: bool
    Enable/disable msi interrupts Default:1, Enable=1 ,Disable=0
  use_msi_x: bool
    Enable/disable msix interrupts Default:1, Enable=1 ,Disable=0

[5y53ng]

Is this the ESX(i) software iSCSI initiator? If so, there are advanced settings on the host for tinkering with the que depth. I only have ESX 3.5 and ESXi 5 hosts at the moment, but go to the configuration tab > advanced settings and you should see some settings in there for the que depth.

[pirx4711]

I did not find the parameter, but I' m also thinking about activating Storage IO Control (SIOC).