topic Re: HP-UX Performance Issue in Operating System - HP-UX

HP-UX Performance Issue

System Dude_1 — Thu, 24 May 2001 00:07:59 GMT

Hi all,

We are running T600 machine with oracle database ( 7.2.3), 3 GB of memory and 4 CPU. We encounter system slowness during peak. Could anybody help on resolving this issue.
# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 145208 379080 28% 0 - 1 /dev/vg00/lvol2
dev 524288 146596 377692 28% 0 - 1 /dev/vg00/lvol3
dev 4096000 146016 3949984 4% 0 - 1 /dev/vg00/lvol12
reserve - 812064 -812064
memory 2297936 1594416 703520 69%
total 7442512 2844300 4598212 38% - 0 -

Re: HP-UX Performance Issue

Andrew_4 — Thu, 24 May 2001 00:43:18 GMT

You need to try to identify the main bottleneck... by comparing your peak period to a 'baseline' of performance....

When you have this 'baseline' (ie, standard performance during non-peak times).. you can compare the processes, memory/swap, disk i/o, and see what has increased alot....

This will hopefully show you where the user reponse time is being dragged down the most...

Once you've identified this, you need to think about how you could improve the performance of this bottleneck... (by getting faster disks, more memory etc etc...)

Hope this helps !

Andrew

Re: HP-UX Performance Issue

System Dude_1 — Thu, 24 May 2001 01:09:12 GMT

There are so many ways to identify the main bottleneck. As for my case it seems that the memory and CPU is bottleneck. We are waiting for new CPU and memory but for the time being what can i do to at least optimised the server performance. Any idea?

Re: HP-UX Performance Issue

Bill Hassell — Thu, 24 May 2001 01:47:50 GMT

Because so much of your swap space is shown as occupied, it is highly likely that you are memory limited. However, with a 32-bit kernel, you can't add enough RAM to make much difference. Verify that you are paging heavily by looking at page-out rates with vmstat. A much more detailed analysis of what you have running (all processes), amount of LAN activity and number of LAN cards, batch reports that are being run, the design of the database index files, kernel parameters and so on, will be needed for a better answer.

The T600 is fairly old technology and it appears that you may have hit the limit of this processor and the load you require. An A-class or L-class will remove address limits and move processing power to today's standards. Alternatively, you can move unnecessary applications to other systems to relieve the load.

Note that the most significant performance changes can be made with better database design and configuration, much more so than working on HP-UX.

Re: HP-UX Performance Issue

Andrew_4 — Thu, 24 May 2001 01:50:59 GMT

You could consider tuning buffer cache... that might alleviate some memory problems (depending on what type of disk access is being performed)...

What is your dbc_max/min_pct set to ??

If you've got max set very high, then the buffer cache will take heaps of memory, and not free it very quickly..
See Doc ID A4460661 for more details...

Hope this helps !

Andrew

Re: HP-UX Performance Issue

System Dude_1 — Thu, 24 May 2001 01:55:29 GMT

dbc_max/min_pct is set to 50/5. what are the impact if I need to change this dbc_max/min_pct and what are the suggested size to change to. Any other parameter need to fine tuning

Re: HP-UX Performance Issue

Andrew_4 — Thu, 24 May 2001 03:08:51 GMT

From DOC ID : A4460661

D. Tuning the File System Buffer Cache.
The problem has been determined to be either a file system I/O
bottleneck or a memory bottleneck. In either case, adjusting
the size of the buffer cache may resolve the problem. It is
easiest to measure the effectiveness of the buffer cache and
the effectiveness of changes if the buffer cache is of a fixed
size, instead of the 10.x default of a dynamic buffer cache that
ranges in size from 5 to 50% of main memory. The size of the
buffer cache is controlled by the kernel parameters bufpages,
nbuf, dbc_max_pct, and dbc_min_pct.

SAM help can provide more information on these parameters, and
the System Administration Tasks manual has information on how
to rebuild the kernel. Historically, HPUX has used 10% of main
memory as a default buffer cache size.

The goal of the buffer cache is to increase the speed of disk I/O
reads and writes. The trade off for that increased performance is
the usage of a percentage of main memory. After isolating the
resource that is slowing the system, the job is to adjust the
size of the buffer cache to get the best returns for this system.
------------

With 3 gig of RAM, I would say that you could safely reduce your max buffer cache... You'll probably find that your processes are swapping alot because the buffer cache is taking up too much space in memory (if it does actually grow to 50% - which is quite possible)...

I would experiment with reducing the max, to say 20%, and see if that reduces the amount of paging your system does...

There is a memeory management white paper at :

/usr/share/doc/mem_mgt.txt

use this guide for more information on tuning your memory.. But the reducing dbc_max_pct would be the first place to start...

Hope this helps !

Andrew

Re: HP-UX Performance Issue

Joseph T. Wyckoff — Sun, 27 May 2001 17:55:54 GMT

In principle, you may be able to buy some time by shuffling files around...

In a database doing anything but simple reads, your transaction logs are being written to constantly...

Therefore, those transaction logs should be on the fastest possible disk subsytem... (of course it should be reliable too..)

Similarly, if you are doing a large number of writes, your database would benefit from being on 'the fastest disks possible'

Here is the gotcha - you really have to understand your usage patterns... it may be faster to have your databases spread across 3 slower disks, rather then everything on one really fast disk..

Similarly, it is CONCEPTUALLY possible to spread swap over several drives, rather than having a few massive swap volumes... in fact I don't know whether this is really useful on HPUX, possible on HPUX, or useful for you.

Unfortunately... I don't know how to ask/answer the question 'is disk i/o a bottleneck' on HPUX.

Good Luck.

Re: HP-UX Performance Issue

Joseph T. Wyckoff — Sun, 27 May 2001 18:11:53 GMT

Is the issue Oracle performance?

Database performance (in general) can be improved or worsened by using INDEXES.

I frankly know NOTHING about Oracle.

If you are doing many inserts, i.e. a transactional database, then you want FEW, well chosen indexes.

If you are doing primarily reads (datamining) then lots of concise indexes might serve you well.

How queries are written can have a big impact, in principle. Some database engines well re-order or re-write queries on the fly to try and find the best indexes to cover queries, so you may not really want to mess with queries. However, it may be worthwhile to train your user or developer community in what is a 'good query' (for your database.)

In some database engines you may have tables that simply are not allowed to change - and could in principle be marked read-only. If this is possible in your scenario, you may find some performance benefit by marking tables as read-only. By the way, if the table is read only, excess indexes won't hurt anything (well, they will suck up disk space)... you aren't making changes here anyway...

As mentioned elsewhere, moving unneccary services off of the box is a good idea...

If DNS is on this box, as an easy example, install it on another box... you may want to keep DNS on this box during a migration, or permanently, but every DNS query you would move off on to another platform would be 'free CPU cycles.'

Of course getting rid of excess services (in my example DNS) entirely would free up memory too - and it sounds like you cannot add memory, you can only take away demands on memory.

Re: HP-UX Performance Issue

A. Clay Stephenson — Sun, 27 May 2001 20:14:02 GMT

Hi,

As has been suggested you need to reduce your buffer cache. max_dbc_pct at 50% is way too high for a database server. If your are using raw disk I/O or OnlineJFS options convosync=direct,mincache=direct which are in effect the same as raw disk then you should really reduce the unix buffer cache since you are in effect double buffering I/O in both the SGA and the UNIX buffer cache. If this box is primarily a database server, I would forget about dynamic buffer cache and set bufpages to about 300MB (and this is generous) and them measure performance.

As has also been mentioned, your T-box is old and even an A-box would blow it away in terms of performance.

Food for thought, Clay

Re: HP-UX Performance Issue

Mladen Despic — Mon, 28 May 2001 16:00:42 GMT

Hi Jamaludin,

Your 'swapinfo' output indicates that pseudo-swap is enabled. I think Oracle recommends that pseudo-swap be disabled as it competes with the database for locked memory resources.

To disable pseudo-swap, change the value of the kernel parameter swapmem_on from 1 to 0.

Mladen

Re: HP-UX Performance Issue

Mladen Despic — Mon, 28 May 2001 16:12:55 GMT

Jamaludin,

If you disable pseudo-swap, you may want to add another 1-2 Gb of device swap. In fact, you may want to add device swap before disabling pseudo-swap.

Also, if you want to implement a number of configuration changes in order to improve your system performance, I would recommend making one change at a time. After each step, observe or monitor your system performance.

Hope this helps a bit.

Re: HP-UX Performance Issue

Magdi KAMAL — Wed, 30 May 2001 15:33:08 GMT

Salam Alaikoum Jamaludin,

You may consider the following points :

1. If you are runnig OLTP applications and their is a lot of concurrent processes, Your swap space MUST be 3 to 4 times you memory capacity. ( I saw it's not the case on your configurations ). If you increase the swap, please do it on different disks for loadbalancing.

2. If you are not treating flat files and your applications are only doing database access, you MUST decrease the "Dynamic buffer max percentage" to 10 % and "Dynamic buffer min percentage" to 2 % . When this parameter ,db_max_pct, is set to 50 % ( default value from HP-UX installation) this means that the seystem could use 50 % from total RAM ( which will be 2 GB in your case ) in order to do lokk a head reading into memory. But if you are using only database and not flat files, this means that all you are reading and putting into memory is for nothing, and global degradation of the system appears.

3. What's about the size of your Oracle SGA allocated structure. This MUST be big enough to handle concurrent users without needs for swapping.

4. You need also to tune your shmmax, this could decrease the number of shared memory segments needed by HP-UX to handle the Oracle-SGA structure.

5. Oracle block size must be the same like hp-ux page size.

Hope this could help you.

Magdi

Re: HP-UX Performance Issue

Bill McNAMARA_1 — Thu, 31 May 2001 14:02:31 GMT

Performance analysis tools such as
Perfview/ Measureware and glance will allow you to identify if the problem is on disk io, cpu or memory or process. After this you can react accordingly.

Perfview is a free 60 day trial from software.hp.com and is good for time graph comparissons. No other analysis s/w is worth getting. PV is quite complete.

Later,
Bill