BladeSystem - General
1748195 Members
3935 Online
108759 Solutions
New Discussion юеВ

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

 
dbueno
Regular Visitor

Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

Hi all,

After doing a couple of days research and trying firmware and driver updates I run out of ideas on what could be the problem with a Bladeserver BL460c and a mezzanine card nc382m (Broadcom 5709S - 1Gb 2-port).

I've tried with the following combinations:

HP custom ESXi 5.1.0 Update2 5.66.28:

    driver scsi-bnx2i                     2.7a.03.v50.2-1OEM.500.0.0.472560     Broadcom

HP ESXi 5.5.0 1746974-HP-5.72A.3-Jun2014 -

    driver - scsi-bnx2i                     2.78.76.v55.1-1OEM.550.0.0.1331820     Broadcom
    driver - scsi-bnx2i                     2.710.50.v55.1-1OEM.550.0.0.1331820    Broadcom (latest version from VMWare's website, dated 30-Oct-2014)

I've configured the card following the instructions of VMWare for 'Dependent hardware iSCSI' which is the case of the mezzanine card I'm using. I also followed the instructions in 'HP Virtual Connect with iSCSI Cookbook'. I'm just using one of the NICs on that card, leaving the other one unconfigured but I'm planning to use it for Round Robin multipath once I realize why is it so slow on reading. Creating a VM with CentOS minimal, the storage is an iSCSI SAN that is where the VMDK of the VM is stored at, and running the following commands this is what I get:

Writing - from RAM (1GB) to disk:

dd if=/dev/zero of=/test bs=4k count=512k
524288+0 records in
524288+0 records out
2147483648 bytes (2.1 GB) copied, 16.6002 s, 129 MB/s

After this test I create a test file with random info the following way: dd if=/dev/urandom of=/test2 bs=4k count=512k

Reading - from disk to /dev/null

dd if=/test of=/dev/null bs=4k count=4k (only 4k as it will take a very long time to complete)
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 70.1791 s, 239 kB/s

The output from hdparm:

hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   2 MB in 29.53 seconds =  69.35 kB/sec

If you need them, I will share the current VConnect and ESXi network and/or storage configuration, but before jumping into any conclusion, please, be aware that using exactly the same settings except for the mezzanine card which I replaced for a nc553m (Emulex 10Gb 2-port). These are the results:

 

Writing - from RAM (1GB) to disk:

 

dd if=/dev/zero of=/test bs=4k count=512k
524288+0 records in
524288+0 records out
2147483648 bytes (2.1 GB) copied, 18.7712 s, 114 MB/s

 

After this test I create a test file with random info the following way: dd if=/dev/urandom of=/test2 bs=4k count=512k

Reading - from disk to /dev/null

dd if=/test2 of=/dev/null bs=4k count=512k
524288+0 records in
524288+0 records out
2147483648 bytes (2.1 GB) copied, 17.9035 s, 120 MB/s

Also note that I'm using a HP 1/10Gb-F VC-Enet module which limits the downlinks to 1GB so there is no reason for paying such a high price for 10GB mezzanine cards when the VC module is limiting it to 1GB anyway.

 

As the only difference is the mezzanine card, the Emulex is a 'Hardware Independent iSCSI card' and the configuration is slightly different than for the Broadcom. Everything else is exactly the same: same slot, same VConnect modules, same transceivers, same iSCSI SAN, same VM, same volume, same Bladeserver, same ESXi version.

 

I don't know if I could be misconfiguring something or missing an important point. Please, let me know if you need more information or to run any command as this is a testing environment and I'm free to test anything.

 

Thanks!

 

 

P.S. This thread has been moevd from ProLiant Servers (ML,DL,SL) to HP BladeSystem Server Blades. - Hp Forum Moderator

5 REPLIES 5
dbueno
Regular Visitor

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

The problem seems to have been solved by itself. The only change we did was deleteting the shared uplink set, do a direct mapping from the 'network' to the 'Uplink port' (no 'shared uplink'). As we couldn't test it with that configuration (VLAN related) we just deleted that static 'network - uplink' map and recreated the 'shared uplink' exactly the same way as we had it before. Suddenly doing some storage benchmarks our VMs report the following performance (iSCSI without multipath, just one NIC enable of the two that the nc382m provides):

 

Writing:

 dd if=/dev/zero of=/root/test.zero bs=4k count=512k
524288+0 records in
524288+0 records out
2147483648 bytes (2.1 GB) copied, 19.2994 s, 111 MB/s

 

Reading:

dd if=/root/test of=/dev/null bs=4k count=512k
524288+0 records in
524288+0 records out
2147483648 bytes (2.1 GB) copied, 16.486 s, 130 MB/s

 

This makes me think about a probable bug either on the HP 1/10Gb-F VC-Enet Module (running firmware v3.61) or on the mezzanine card (nc382m with latest firmware from SPP Sep2014). Any words from HP on this issue please? We would like to make sure that this problem won't happen again.

 

Thanks!

Renjiv
Respected Contributor

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

Greetings!

 

It seems it was a problem with too much I/O during the backups, and the CPUs of the servers couldn't handle that much I/O for iSCSI. Also this can occur during heavy network load the network traffic might be stalled

 

This issue is resolved with the latest ROMBIOS release.

 

Regards,

Renji V

 



Was this post useful? -
To say thanks click the "thumbs up" icon below!!! .....
dbueno
Regular Visitor

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

Hi Renjiv,

 

Thanks for your contribution. I wasn't doing any backup, just testing the iSCSI performance with the NC382m cards. I'm not sure if I understood regarding the ROMBIOS you're talking about, is that a new BIOS version for the Proliant BL460c? Is it for the NC382m mezzanine card? Or maybe for the HP 1/10Gb-F VC-Enet module?

 

If you could provide a link for that ROMBIOS I would be very open to test it again, but as I said before, nothing has changed besides deleting the 'shared uplink' and the 'network' on the VCM and recreating it again. That is where it points to a firmware problem as with just a configuration modification, keeping exactly the same settings, the performance has also changed.

 

Renjiv
Respected Contributor

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

Hello,

 

Yes, I am referring to System BIOS latest version.

 

Could you please let me know which model sever you have (eg: BL460c G6, BL460c G7 or BL460c Gen8 etc..)

 

Also please use the below URL, then enter the correct model # of the server. Then Select theOperating System installed and selct the BIOS to download the latest BIOS.

 

http://h20564.www2.hp.com/hpsc/swd/public/readIndex?sp4ts.oid=1842756

 

Regards,

Renji V



Was this post useful? -
To say thanks click the "thumbs up" icon below!!! .....
dbueno
Regular Visitor

Re: Very slow speed issues in ESX 5.1/5.5 with BL460c and nc382m in iSCSI

Hi Renji,

 

Thanks for your reply. The Blades we have are BL460c G6 and they've been updated with the latest BIOS/firmware of the HP Service Pack for ProLiant (SPP) Version 2014.09.0. Anyway, it is running 'ROM Version I24 10/01/2013'

 

The problem reappeared again on the same c7000 bladechassis with the same componets (VConnects, Mezzanine cards, same bladeservers). This performance issue also happened to us on a different c3000 bladechassis we also have for testing.

 

Please, let me know how to proceed to gather as much information you'll need to debug this issue.

 

Thanks!