StoreVirtual Storage
Showing results for 
Search instead for 
Did you mean: 

MAssive performance issues after SANIQ 9 installation!

Thomas Mill
Occasional Contributor

MAssive performance issues after SANIQ 9 installation!

Hi folks.

20 node, p4000 SANIQ 9 environment (Running primarily Exchange 2007 with HP servers and ESX).

We do not use MPIO as we had serious issues with it when we tried to use it 30 months ago (We have zero history of i/o performance over that time). We also do not use "Flow Control". Both "Flow Control" and MPIO have never been used by us in over 2 years, yet are being advised as possible "reasons" for our problems since SANIQ 9!!

Effectively, we upgraded all 20 nodes to saniq 9.0 on Tuesday 15th March, and that evening our backup performance dropped from 40 mb\sec, to about 12mb/sec...and this is still the case now. In addition, performance of ALL applications that use this storage are strongly affected whenever a resync takes place on the the array..

Has anyone experienced anything like this, or have any suggestions as to what we can do. Vendors are coming onsite to assist us, but it irritates me that they are suggesting enabling "flow control" and MPIO as ways to address this, when they should be working with us to find out what caused this to happen in the first place. MPIO and FLOW Control creates a great deal of prep work and testing for us which we will take on board later, ONCE they determine why SANIQ 9 has caused this problem! We also upgraded all nodes to RAID driver 7.22 the day after, but the problems were already apparent the night before.

Any advice out there?
Occasional Advisor

Re: MAssive performance issues after SANIQ 9 installation!

Thomas, not sure about advice but can share our upgrade nightmare with you;

We upgraded to v.9 on a 12 node SAS cluster (p4000), the day after we experienced issues with;

Erroneous alerts from nodes with disks uninitialized, SNMP server down, not formatted and not ready alerts as well as storage overloaded issues with one node,

We primarily experienced a node overloading for about 5 minutes, then all volumes resyncing every couple of hours, while this happened our SQL clusters with DSM installed blue screened.

HP suspected a disk fault, 2 reboots did not fail a disk. We had a level 2 support look at the logs and identified a drive which had not failed but showing long latency times. We replaced the disk as per his recommendations and voila! no more errors at all.

One other thing to note is that we uninstalled the DSM while this was going on and SQL clusters stayed up after that which was rather confusing.

The Joys of upgrading!
Valued Contributor

Re: MAssive performance issues after SANIQ 9 installation!

Hello what is your performance on a lun in this enironement with 20 - 12 nodes.
Do you have alot of small luns or a few big once, because we got stuck @ arround 40 MB/s