K400 Performance - What next?

[Russ Hancock_1]

I've got 4 processors running at around 10%,
1Gb memory that's never swapping out, and 12 disks that very rarely go above 30% busy, and yet this box still runs really slowly.

I've changed a few kernel parameters, timeslice (to 10) and bufpages (to 25600), these seem to have helped, but I'm now at a loss where to look next.

Is it possible that the SCSI cards could be configured wrong? What should I be looking for in the SAR results...

Cheers
Russ.

[harry d brown jr]

Russ,

Use glance to get a true view of your system. sar, top, uptime, vmstat, and iostat won't give you the info you really need. What OS level are you running, and has it been brought up to date in patches (sept 2001)? Also, what kind of processing goes on the K? Do you have an oracle db? What kind of processes are the top consumers of CPU?

live free or die
harry

[Roger Baptiste]

>

What do you mean by - really slowly?? Are the applications response time slow?? Is any unix command responding slowly? Where exactly are you seeing the problem of slowness? Performance problem is a tricky thing - the key is start of defining where exactly you are seeing the problem.

>

Timeslice is a parm better left untouched. Regarding bufpages, you would need to look the sar -b ouput to decide how the buffer is performing. (look at read and write cache hit rate). Also, it is prefreble to have it set as dynamic ranging between 10 and 5% - max to min.

< but I'm now at a loss where to look next.>>

Check glance and see where the bottleneck is. Is the CPU usage high? Is the Memory usage high? is there disk bottleneck showing up on the glance screen? How is the swap usage (swapinfo -mt ).
The key is to reduce the problem to a specific unit
and then take it from there.

<< Is it possible that the SCSI cards could be configured wrong? What should I be looking for in the SAR results...>>

do sar -d and look at the avserv and avwait times.

also do vmstat and see the pageout (po) values. It should be ideally 0 .

HTH
raj

[Sanjay_6]

Hi Russ,

Take a look at this thread,

http://us-support.external.hp.com/iv/bin/doc.pl/sid=05cf21d414ed6ba3d0/screen=ivHome/?NODEID=English_SHW::WW_SW_UX_SYSADM_EN_E/Q1.7&WARP=1

Hope this helps.

Regds

[A. Clay Stephenson]

Hi Russ:

If I were you at this point I would install the trial version of Glance from your Application CD's. Using gpm you should be able to narrow down problems (if they exist). Be aware that there are a few pathological cases of multi-threaded software that actually run worse on multi-processor boxes than on single processor boxes because of constant TLB flushing. I doubt this is the case. I suspect a crucial disk bottleneck now. By the way, the bufpages at 25600 was meant to be a starting point. You then use sar or glance to start fine-tuning. I am a very strong advocate of non-dynamic buffer cache in most cases; it is much easier to tune other values when buffer cache is fixed.

[Russ Hancock_1]

Hi Sanjay,

I can't follow that link, is says I'm not authorised? Could you attach a copy for me please..

Cheers
Russ,.

[Russ Hancock_1]

Guys.
We are running HPUX 11, and I will check on the patching levels, I doubt whether it's been patched within the last 2 years!!. This box is used mainly as a Oracle Database, with the top processes being oracle related, but these never use more than 1-2% of the CPU. It also appears that Glance was installed on this machine last year sometime, can it be installed again as a demo?
By responding really slowly, I mean simple copy commands, bdf, ll -rt that sort of thing but not on any specific disks.
Like I said before the CPU usage, and Memory are both very low, using the sarcheck demo..
Average CPU 9%, Average swapout rate 0/sec.

What's a good figure for avserv and avwait?

All my perceptions of fast and slow are based on our old K100, it seems much quicker at the simple commands even when Oracle isn't running on this server.

Thanks for the pointers.
Russ.

[Sanjay_6]

Hi Russ,

Take a look at this thread,

http://docs.hp.com/hpux/onlinedocs/os/11.0/tuningwp.html

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2355-90701/B2355-90701_top.html&con=/hpux/onlinedocs/B2355-90701/00/00/57-con.html&toc=/hpux/onlinedocs/B2355-90701/00/00/57-toc.html&searchterms=system%7cperformance&queryid=20011126-081027

Hope this helps.

Regds

[Bill Hassell]

>> What's a good figure for avserv and avwait?

Your system is essentially idle. There have been a number of patches to address delays internal to the opsystem for 11.0--patches are a requirement before dealing with any performance issues. Current patch bundle is Sep-2001.

avserv is a meaningless metric, left over from the good old days when the disk was a simple (virtually no on-board cache) device and there were no issues about multiple I/O's pending on the disk bus. sar (nor Glance or other measurement tools) cannot practically measure the start-to-finish time for every I/O with drivers that handle multiple channels to a single disk and mirroring, etc, (the metric would be very costly in CPU overhead), so you can ignore that value in sar.

avwait in low 10's and 20's is fine but recognize that a sudden burst in the hundreds can be normal.

As you will see from the tuning documents, this is not simple (very difficult without a modern tool like Glance). And once Glance has been installed as a trial version, it must be purchased to install it again.