Email Subscription Notifications Suspended Temporarily
We are in the process of making navigation in the Servers and Operating Systems forums simpler and more direct. While doing this, we have to temporarily suspend email notifications for subscriptions. If you are subscribed to one or more discussion boards or blogs in the community, please check them daily to see new content. Notifications will be turned back on in a few days. We apologize for any inconvenience this may cause. Thanks, Warren_Admin
StoreVirtual Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

After upgrade to SAN IQ 9.5, write peformance issue

Johnisccp
Occasional Contributor

After upgrade to SAN IQ 9.5, write peformance issue

Since I upgraded our HP LEFTHAND 4500 to SAN IQ 9.5, we are experiencing write delay.  I did not expperience this issue on 9.0.  I have flow control on the HP Lefthand 4500, the hosts and the switches.  Has anyone experience this issue?

15 REPLIES
vStorage
Occasional Visitor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Yes, SAN/iQ 9.5 does appear to have measureable write latency issues under vSphere 5 that previous versions did not

 

What hypervisor/OS are you running, what switch type?

 

Where are you seeing the latency? OS or in the CMC itself?

 

 

Couchman
Frequent Visitor

Re: After upgrade to SAN IQ 9.5, write peformance issue

I'm getting increased latency too on our P4300 nodes upgraded from 8.5 to 9.5. Both reads and writes are worse than before but particularly writes. Currently I have a call open with HP but they are taking them time about it.

I'm seeing the latency in the vSphere client and also in the CMC

Did you get your sorted out John?
oikjn
Honored Contributor

Re: After upgrade to SAN IQ 9.5, write peformance issue

I"m fully patched and not really seeing any write delay issue that I've noticed, but I"m running 100% M$ w/ the HP DSM and nothing on VMware. 

5y53ng
Regular Advisor

Re: After upgrade to SAN IQ 9.5, write peformance issue

I have observed the same problem with write latency using SANiQ 9.5 and vSphere 5.

Russell Couch
Occasional Visitor

Re: After upgrade to SAN IQ 9.5, write peformance issue

How much latency are people seeing?

 

VMware is showing some fairly extreme latency here, one low use VM averaged 35ms over a couple of hours. The CMC shows about 5.5ms over a 24 hour period for the entire SAN which doesn't sound that bad, but I have no CMC reference benchmarks to look at. My VMware benchmarks show there is obviously a problem.


Cheers

Russ..

twg351
Occasional Advisor

Re: After upgrade to SAN IQ 9.5, write peformance issue

I have experienced the same issues.  Although I use HyperV R2 w/ HP Procurve 2190al switches.

 

I originally had 2 P4300 units on v9.0.  When I bought 2 additional P4300 units and added to the cluster, I had to upgrade all the v9.5 (the version that came with the 2 new P4300's).  After the upgrade to v9.5 I have experienced a READ/WRITE slowdown.  Nightly backups & IOMeter are about 25% slower for File Server, and about 33% slower for Exchange.

 

Luckily my speeds were good enough originally that we are living with this.  But I was bummed as I expected a speed increase when I added 2 additional nodes.  

 

So it's not just VMWare or HyperV.  It's v9.5 ... and yes I am fully patched as a a few weeks ago.  haven't bothered to check since then.

ryan_1212
Advisor

Re: After upgrade to SAN IQ 9.5, write peformance issue

I was planning my upgrade from 9.0 to 9.5 next weekend.. I have a 4 node P4300G2 with 55 VMs. Needless to say my environment will not function if storage gets any slower. I already have minor performance issues.

 

What is HP saying about this?

 

Hopefully not their usual course of action- blame it on your network and waste hours running iperf tests between your nodes and telling you that because one out of 15 tests showed bandwidth of 600mbps instead of values over 900mpbs that it is your networks fault, even though your NSMs network traffic as measured in the CMC is like 150mpbs?

Couchman
Frequent Visitor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Yeah thats pretty much it so far but ours is pretty clear cut in the stats since the upgrade. They first blamed it on a couple of disks which they claimed to be faulty that had no sign of errors, then I must have returned a 4 or 5 sets of performance logs and support bundles after that. Now they want me to shutdown the whole SAN and VM environment as is sometimes fixes the issue apparently...not falling for that one.

 

We're running synchronous multi-site and personally I think 9.5 has some kind of extra multi-site latency introduced to cover for some past issues, maybe it needs to write to both sites before the write is confirmed?

 

Anyway, I have some local HP guys involved and the call is now at level 3 so hopefully some kind of resolution or statement from them soon. Maybe even a remote session to confirm the issue might be nice.

 

For now I wouldn't upgrade to 9.5 as even our new 2 node P4500 cluster is showing the same issues.

Ryan Nelson_3
Occasional Contributor

Re: After upgrade to SAN IQ 9.5, write peformance issue


Couchman wrote:

We're running synchronous multi-site and personally I think 9.5 has some kind of extra multi-site latency introduced to cover for some past issues, maybe it needs to write to both sites before the write is confirmed?

 

I was under the impression that is how writes worked with sync the multi-site setup. 

oikjn
Honored Contributor

Re: After upgrade to SAN IQ 9.5, write peformance issue

it is...  it has to be that way...  if you are synchronous the write doesn't "count" until it exists at both sites.  That definitely adds to latency since the write has to go to both sites and be confirmed at both sites, but that is the definition of synchronous... otherwise it would be asynchronous.

Amana Living
Occasional Advisor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Hi, is this still an issue? I was thinking of upgrading our SAN's from 8.5. Should I just go to 9.0 then?

Besty_Boy
Occasional Contributor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Hi, I have just upgraded our multi site san from from v9 to 9.5 patch set 3 without an issue. We had to reduce our volumes to under 2TB as we were having performance issues while the nodes were resynchronising before hand. Once completed everything went very smoothly. I have had no reported performance issues.
Couchman
Frequent Visitor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Yes we still have a problem, a case is still open which is now with the HP VMware team as the storage team think everything is OK with the SAN. Are you running asynchronous multi-site configuration Besty_Boy?

Besty_Boy
Occasional Contributor

Re: After upgrade to SAN IQ 9.5, write peformance issue

We are running a 6 node hyper-v synchronous multi-site SAN. This is split over 3 site with the quorum on the 3rd site. The distance between storage nodes is about 2km.
David_Tocker
Regular Advisor

Re: After upgrade to SAN IQ 9.5, write peformance issue

Yes but what is the latency between the sites? it should be bugger-all for good performance.
2km tells us nothing. How are they linked? Bandwidth?
I only run a single site cluster, but as far as i can tell 9.5 has not affected the performance (SCOM logging confirms this)
Regards.

David Tocker