Sucky Performance after XP-512 firmware upgrade?

[Joe Bruno]

Well, application owners are claiming 50% increase in processing time for sql programs.

Im seeing very high system cpu user and very little user cpu.

I've attached a sar output. Can someone give me their thoughts? This is an RP8400 running oracle. HP 11.11

[Bruno Vidal]

Hi,
I've been really often involved in XP escalation performance, and i can say you that you have really good performance with your XP, all avser and avwait are both < 5ms (it is what you can have with an XP512). You have only two disk that have measure at 10ms, but it is still acceptable (perhaps take a look at which lvol use it, and if it is redo log for example, then try to spread it over multiple luns).
So I think you should look somewhere else. If you are using 11.11, and that you have an hight cpu usage, I think that you should take care to your buffer cache utilisation. I know that with database, buffer cache is of no use, but your data are still going throught BC (you can avoid this by using new functionnality in JFS 3.5 Direct IO). But I think it would be a good idea to install:
PHKL_27808 11.11 Filesystem buffercache performance fix
And that you change the new kernel parameter bcvmap_size_factor to 8. Normally it must really improved your BC performance (in case of saturation of the bcv map).

Cheers.

[Michael Steele_2]

Looks good to me.

Disk bottlenecks aren't an issue, low %busy.

I/O bottlenecks aren't an issue, with 'avwait' or 'avserv'.

(* Bad 'avserv' is 20ms or higher. *)

What about:

sar -v 5 5
sar -u 5 5
vmstat 5 5
swapinfo -tam

Also check the I/O balance out the HBA's:

ioscan -fnC fc

...and note the /dev/td# devices. Now,

fcmsutil /dev/td0
fcmsutil /dev/td1, etc

Determine the number of I/O transactions and reads and writes. If one /dev/td adapter is grossly large and the other is a fraction of the other then all of you reads and writes are going out this device and not the other.

[Joe Bruno]

Maybe I didn't really sample the system during its heaviest times. See attached. I've ran the other commands you listed.

[Bruno Vidal]

Hi,
The strange thing is that you have a really hight value for sys. It means that your system is spending lots of time to do something. I think you are really hitting a bottleneck inside the kernel (a system table too small for example). But it is not a disk problem as far I can see. Could you try some cp of big file, just to ensure that BC is not your bottleneck.

Cheers.

[Joe Bruno]

I guess I really need help trying to determine why system CPU is so high. Any suggestion. We dont use Business Copy

[Steven E. Protter]

We had a system that suddenly slowed down and starting showing bottlenecks.

One of the CPU's had decided it needed a summer vacation.

the GSP command ss showed that the processor had unconfigured itself. In English the darned thing had burned out.

There was no EMS message, no other warnings. The good news is the system kept running.

Is it possible that a CPU went at the time or soon after the upgrade?

SEP

[Michael Steele_2]

Ok, %wio is high and this indicates a disk or tape bottleneck. CPU is fine as is everything else.

Need that 'fcmsutil' info.

Continue sampling at high load periods with :

sar -u 5 5
sar -d 5 5

Look into glance advisor or use a cron.

Got any tape devices running?

[Tom Williams_3]

In this context, "BC" meant "buffer cache", not Business Copy.

We recently tuned our rp8400 / XP512 connection, and the following week had absolutely horrid performance because the AutoPath XP policy was not properly restored after a reboot -- so we were using one path for all I/O. The autopath retrieve kept complaining that the array configuration had changed, so we wrote a quick-and-dirty script to force the load balance policy at every boot.

During that time, our system was so bad it took 20 seconds for a bdf, a minute to log in, and batch took 3 times as long.

Now, our performance is MUCH improved with our 6 paths, but we still have occasional pockets of high avwait (> 100!) that pop up and go away. We're applying PHKL_27808 tonight (that's how we found this thread) - I'll let you know how it goes for us.

[Tim D Fulford]

What is weird looking at it is that most disks/LUNs on c41 are faster than c32!! they are also being used less!

I guess you are using even LUNs down C32 & odd down c41. I think the XP512 has a similar layout to VA74xx & you have two redundacy groups, one controled by ctl1 & other ctl2. If this is so, are you sure the controller path to xp512 to rundundacy group are properly alligned? If not then information will be going over the xp backplane which will not help!

Regards

Tim

[Tim D Fulford]

Oh I forgot... roll back the firmware upgrade if it is that bad?!!!

Tim