topic Re: Virtual Connect Latency to EMC SAN in BladeSystem - General

Virtual Connect Latency to EMC SAN

Entertainer — Thu, 24 Jan 2019 01:26:45 GMT

We've been having high disk latency between C7000 Blade enclosures hosting BL460c to EMC SAN using hardware iSCSI within Virtual Connect on VMware ESXi. This appears to be an issue on 6.0, 6.5, and 6.7 so we've eliminated VMware as the problem. Latency goes away between DL360 and EMC, seems isolated to CClass using Virtual Connect. Wondering if anyone has seen this or has ideas how to fix it?

Re: Virtual Connect Latency to EMC SAN

Siddhartha_M — Mon, 28 Jan 2019 09:47:28 GMT

Hello There,

There is no detailed information provided, we need to understand the tpology of your enveiorment.

We also need to understand where the packet lost is hapenning.
Since there are multiple technology involved and no staright forward answer to your query hence would suggest to log a support ticket on a valid s/n to investigate the issue further.

I am an HPE Employee.

Thanks,

Siddhartha_M

Re: Virtual Connect Latency to EMC SAN

Entertainer — Tue, 29 Jan 2019 16:30:22 GMT

I'll open a case. For the sake of argument here's the scenario for you: I have all the equipment above in place with the exception of the HP CClass chassis. ESX hosts running on Cisco Unity blades. When I hooked up the CClass the latency started. If I shut down the blades on the CClass the latency stops. Boot the Blade up, latency again. I think this eliminates most of the equipment above.

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Tue, 29 Jan 2019 17:16:21 GMT

Hi

Did you install latest firmware/drivers ? there are some issues like that in
previous versions also

May be sure to be at least to VC 4.50 firmware

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

Entertainer — Tue, 29 Jan 2019 17:19:58 GMT

We're already on Firmware version 4.62.

Thanks for the suggestion though.

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Tue, 29 Jan 2019 17:27:21 GMT

Check firmware HBA

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

Alok2019 — Thu, 31 Jan 2019 11:14:51 GMT

This is a very burning issue, already 400+ view but no useful record seen. Can some one share learning or resolution on this.

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Thu, 31 Jan 2019 11:19:13 GMT

In most of case firmware update fix this kind of issue, if not please open
case to HPE support

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

Entertainer — Thu, 31 Jan 2019 13:13:13 GMT

Not sure what you mean by HBA firmware. I've updated all associated firmware to the latest released.

Re: Virtual Connect Latency to EMC SAN

Entertainer — Thu, 31 Jan 2019 13:14:20 GMT

I have opened a case with HPE support and am currently talking to L2 Virtual Connect support. I will update you all with the conclusion of that case.

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Thu, 31 Jan 2019 13:19:13 GMT

Fine, what are San switch ? (Cisco, Brocade ?)

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

Entertainer — Thu, 31 Jan 2019 13:25:33 GMT

Cisco Nexus

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Thu, 31 Jan 2019 13:57:13 GMT

What cisco logs are seeing ? during latency ?

Something like that ?

https://www.cisco.com/c/dam/en/us/products/collateral/storage-networking/mds
-9700-series-multilayer-directors/whitepaper-c11-737315.pdf

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

sgunelius — Thu, 07 Feb 2019 13:24:17 GMT

Which generation BL460c? What FLB adapter/mezzanine are you using in the BL460c? Which VC module(s) are you using in the c7000 and are the enclosures Gen1, Gen2 or Gen3? Did you get any where with HPE technical support?

We're using BL460c Gen9 with 650FLB adapters and VCFF 20/40 F8 modules (v4.63) in c7000 Gen3 enclosures, but I'm using FCoE/FC to connect to our Nexus 5672UP-16G TOR switches and upstream to our arrays.

Re: Virtual Connect Latency to EMC SAN

Entertainer — Thu, 07 Feb 2019 13:38:01 GMT

Hopefully I catch all of these questions. First, this is happening on BL460c gen 8-10. The generation doesn't seem to matter. We're using C7000 chasis. They're less than 2 years old so I would assume the latest generation although I don't remmber there being generations of C7000s. One of them is literally brand new. The VC modules are the same, HP VC Flex-10/10D Modules. I'm guessing they're different generations, but all the same model. The FLB adapter type is an Emulex, but outside of that I'm not sure. We've tried various firmware as well on them.

My current support case with HPE has been escalated to Level 3 - engineering. Currently no resolution has resulted. Waiting for them to go through diagnostics.

Re: Virtual Connect Latency to EMC SAN

jdgs — Tue, 21 May 2019 08:23:03 GMT

Does anyone have any update on this? We are seeing an issue which seems to be similar - may or may not be the same thing.

EDIT for clarity:

We are running C7K with Virtual Connect Flex-10 into a Cisco Nexus 5K stack. We have SANs with multiple vendors, both delivered as NFS and iSCSI.

We are consistently seeing high read latency much above expected levels, however write latency is very low as expected.

Re: Virtual Connect Latency to EMC SAN

Entertainer — Tue, 21 May 2019 11:26:04 GMT

HPE never responded to this thread. We are still seeing this issue. I've opened a case with HPE, Cisco, and EMC and have been troubleshooting for months but am at a point where what HPE are asking me to do is unreasonable and I'm not able to go forward. Per a suggestion from Reddit, I've swapped out some blades for a few DL360 but the problem persist. We are unable to determine if the issue is caused by the blades/CClass/FlexConnect and is being expanded to the DL series, or if there is a problem elsewhere.

Another curiosity is that we installed a CCLass in a different datacenter that was running all Cisco blades. As soon as we connected the blades to the SAN from with in the OS, latency appeared. To clarifiy; same SAN, networking, LUNs, OS. Only differnce was HP blades instead of Cisco. Something is definitely wrong here but it's very hard to pinpoint. I'm leaning heavily towards CCLass/FlexConnect as the issue. Let me know your outcome if you determine a solution.

Re: Virtual Connect Latency to EMC SAN

frenchy94 — Tue, 21 May 2019 11:39:45 GMT

What is OS ?

What is Blade type including HBA ?

What is Virtual connect Type ?

If you have same issue with rack server, i can say that is not an issue with
blade architecture

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

Re: Virtual Connect Latency to EMC SAN

Entertainer — Tue, 21 May 2019 11:55:36 GMT

We went over this already earlier in the post. Additionally, it most certainly can be a flex connect issue if the flex connect is causing a problem on the network and/or san that is being seen elsewhere.

Re: Virtual Connect Latency to EMC SAN

jdgs — Tue, 21 May 2019 14:17:58 GMT

If you have some blades running, and then you add a DL, do you get the same poor performance on the DL as well, or just the poor performance and latency exists on the blades? I get that the read latency reports poorly on the array, but if. for example you run a copy operation from a VM on the DL, do you see the same performance as on the BLs, or is performance much better?

Another question - how does you write latency look vs your read latency?

The reason I ask is that I am considering doing the same thing. We have a situation that sounds very similar to yours:

HP BLs (Gen 8/9), C7K, VC Flex 10, Nexus 5K, and WDC Tegile SAN (NFS).

What we are seeing is excellent network latency (sub 1ms), excellent write latency to storage (sub 5ms), but read latency fluctuates anywhere upwards of 50 or even 100ms at times, causing obvious performance issues.

The storage vendor has been troubleshooting this for an extended period but cannot get to the bottom of it. Additionally, we have had our networking review the Nexus and confirm everything appears as it should be.

My next step tomorrow was to introduce a DL to see if this suffers with the same poor read issues.