After upgrade to SAN IQ 9.5, write peformance issue

[Johnisccp]

Since I upgraded our HP LEFTHAND 4500 to SAN IQ 9.5, we are experiencing write delay.  I did not expperience this issue on 9.0.  I have flow control on the HP Lefthand 4500, the hosts and the switches.  Has anyone experience this issue?

[vStorage]

Yes, SAN/iQ 9.5 does appear to have measureable write latency issues under vSphere 5 that previous versions did not

What hypervisor/OS are you running, what switch type?

Where are you seeing the latency? OS or in the CMC itself?

[Couchman]

I'm getting increased latency too on our P4300 nodes upgraded from 8.5 to 9.5. Both reads and writes are worse than before but particularly writes. Currently I have a call open with HP but they are taking them time about it.

I'm seeing the latency in the vSphere client and also in the CMC

Did you get your sorted out John?

[oikjn]

I"m fully patched and not really seeing any write delay issue that I've noticed, but I"m running 100% M$ w/ the HP DSM and nothing on VMware. 

[5y53ng]

I have observed the same problem with write latency using SANiQ 9.5 and vSphere 5.

[Russell Couch]

How much latency are people seeing?

VMware is showing some fairly extreme latency here, one low use VM averaged 35ms over a couple of hours. The CMC shows about 5.5ms over a 24 hour period for the entire SAN which doesn't sound that bad, but I have no CMC reference benchmarks to look at. My VMware benchmarks show there is obviously a problem.

Cheers

Russ..

[twg351]

I have experienced the same issues.  Although I use HyperV R2 w/ HP Procurve 2190al switches.

I originally had 2 P4300 units on v9.0.  When I bought 2 additional P4300 units and added to the cluster, I had to upgrade all the v9.5 (the version that came with the 2 new P4300's).  After the upgrade to v9.5 I have experienced a READ/WRITE slowdown.  Nightly backups & IOMeter are about 25% slower for File Server, and about 33% slower for Exchange.

Luckily my speeds were good enough originally that we are living with this.  But I was bummed as I expected a speed increase when I added 2 additional nodes.  

So it's not just VMWare or HyperV.  It's v9.5 ... and yes I am fully patched as a a few weeks ago.  haven't bothered to check since then.

[ryan_1212]

I was planning my upgrade from 9.0 to 9.5 next weekend.. I have a 4 node P4300G2 with 55 VMs. Needless to say my environment will not function if storage gets any slower. I already have minor performance issues.

What is HP saying about this?

Hopefully not their usual course of action- blame it on your network and waste hours running iperf tests between your nodes and telling you that because one out of 15 tests showed bandwidth of 600mbps instead of values over 900mpbs that it is your networks fault, even though your NSMs network traffic as measured in the CMC is like 150mpbs?

[Couchman]

Yeah thats pretty much it so far but ours is pretty clear cut in the stats since the upgrade. They first blamed it on a couple of disks which they claimed to be faulty that had no sign of errors, then I must have returned a 4 or 5 sets of performance logs and support bundles after that. Now they want me to shutdown the whole SAN and VM environment as is sometimes fixes the issue apparently...not falling for that one.

We're running synchronous multi-site and personally I think 9.5 has some kind of extra multi-site latency introduced to cover for some past issues, maybe it needs to write to both sites before the write is confirmed?

Anyway, I have some local HP guys involved and the call is now at level 3 so hopefully some kind of resolution or statement from them soon. Maybe even a remote session to confirm the issue might be nice.

For now I wouldn't upgrade to 9.5 as even our new 2 node P4500 cluster is showing the same issues.

[Ryan Nelson_3]

---

@Couchman wrote:

We're running synchronous multi-site and personally I think 9.5 has some kind of extra multi-site latency introduced to cover for some past issues, maybe it needs to write to both sites before the write is confirmed?

I was under the impression that is how writes worked with sync the multi-site setup.