Performance problem postmortem

[Jeff_Traigle]

We had a performance issue on one server last week that lasted about 2 minutes. The metrics extracted from Measureware seem to indicate a disk or disk subsystem bottleneck.

Peak Time (GLB_DISK_UTIL_PEAK?) shot up to 67.56 when normally in the single digits or teens.

Pk Dsk Disk (GBL_DISK_UTIL_PEAK?) shot up to 202.882 when it's usually in the teens or 20s.

Pri Prompt (GBL_PRI_QUEUE?) went to 19.96 when it's usually near 0.

Load average went up to 3 at the time (as reported by the by non-sysadmins who were monitoring application performance).

DskSubsys User (GBL_DISK_SUBSYSTEM_QUEUE?) went up to 0.19 for the 5 minute interval prior to the reported time of the performance problem and was 0.04 for the 5 minute interval of the performance problem. This is normally 0.

Unfortunately, no detailed disk information appears to be available from Measureware for that entire day.

All other metrics look reasonable.

We're also using Secure Path (A.3.0F.03F.00F) on this system, in case that comes into play. The LUNs are configured to use Shortest Queue Length as the load balancing policy. (Not sure this is the best one to be using even.)

We had an abnormal workload dumped on the system at the time from a customer, but I don't know the details of that. It would not be the first time we've gotten hammered by a customer, but that usually manifests itself as a CPU bottleneck. Of course, without knowing the profile of the application running at this instance, it's difficult to determine too much on generalities.

Unfortunately, we also were running a synchronization of mirrorclones on our EVA 6000s while this performance problem occurred. That process was started 5 minutes prior to the reported performance issue and continued for more than 45 minutes after the incident ended. That process is still being blamed for the problem despite it being a very unlikely candidate. I'm not sure how to collect any historical data (if any is even available) from EVAperf since we're running CV EVA 8.0.2, which does not seem to have the "hist" command that is mentioned in the CV EVA 9.1 User Guide.

So, the following questions come to mind immediately for anyone who knows better than me...

1. Is there a way to see any historical data regarding EVA performance from CV EVA 8.0.2? (We can't upgrade to 9.1 until we get firmware upgraded.)

2. Is there anything else we can look at that might shed more light on what was happening?

This is on HP-UX 11.11 so maybe it has something to do with how it handles disk I/O? (e.g. Even though the EVA could have handled everything the server wanted to throw at it, the OS throttled the I/O flow, causing a bottleneck unnecessarily?)