topic Re: SAN IQ 9.0 performance counters in StoreVirtual Storage

SAN IQ 9.0 performance counters

JLF175_1 — Thu, 16 Dec 2010 16:25:15 GMT

Hi,
I'm noticing something odd after installing the 9.0 software for my two node P4300 Starter SAN.

When viewing the basic stats on the Performance Monitor the Queue Depth Total for the cluster is being reported as 2,155. That's obviously quite high.

so, I broke it down by adding the latency and queue depth total for all the datastores on the cluster. The queue depth for some datastores was showing near zero (as past data shows it should be) but a few were in the hundreds including a new datastore which is only holding VMWare template files so there should be little to no activity on it.

The latency readings for all datastores is hovering around zero as well.

There's also two other pieces of interesting information. First, the Storage devices themselves show queue depth totals of zero and latencies of near zero.

Also, the Queue Depth Totals that are high are staying consistently high with very little fluctuation so their graphical line is dead straight.

so, performance is fine but the Queue Depth Total numbers suggest it shouldn't be.

I'd appreciate any help on this matter.

Thanks
Jamie

Re: SAN IQ 9.0 performance counters

DMcCoy_1 — Fri, 17 Dec 2010 13:22:25 GMT

I have the same issue. Also P4300G2 starter.

Currently trying to get support to recognise it as an issue. The monitor seems to be increasing the value during load, and then re-adding the value to the previous one when the load drops instead of subtracting this amount. It makes the queue depth monitor completely useless, I did not see this under 8.5.

Re: SAN IQ 9.0 performance counters

JLF175_1 — Fri, 17 Dec 2010 13:48:47 GMT

I think you're right. The counter for my Queue Depth Total is now sitting at nearly 11,000, most likely due to backups last night.

It appears the system is reporting the summation of all queue depth rather than a real-time number.

Re: SAN IQ 9.0 performance counters

teledata — Fri, 17 Dec 2010 14:34:44 GMT

If it is creating a running total it sounds like they assigned it a "counter32" type instead of "gauge"...

If you are using a 3rd party monitoring with SNMP you can temporarily fix it by changing the measure type from gauge to counter...
Or wait for HP to patch the software ;)

Re: SAN IQ 9.0 performance counters

Fred Blum — Fri, 17 Dec 2010 22:21:43 GMT

I am also running SANiQ 9 after upgrading from SANiQ 8.5 but on my system it isn't a running total.
Looking at it while a database backup is performed Queue Depth Total value mostly 0 (changing), max 2, average 0.2 changing now to 0.1.

I am running DSM 9 on the server nodes.

Re: SAN IQ 9.0 performance counters

Paul Hutchings — Sun, 13 Feb 2011 20:00:41 GMT

Did anyone confirm for sure that this is an issue?

I'm sitting looking at a lightly loaded P4500 cluster with a queue depth of 14,650 if the CMC is to be believed.

Re: SAN IQ 9.0 performance counters

chadw_1 — Mon, 14 Feb 2011 16:54:09 GMT

I am having the same issue and have just disabled the queue depth counter. Hoping it is not a "real" issue.

Re: SAN IQ 9.0 performance counters

Paul Hutchings — Tue, 15 Feb 2011 10:03:33 GMT

Known issue and to be fixed in an upcoming CMC update according to HP technical - date of CMC update not known.