Performance Monitoring advice please

[Tony Flaherty]

We have an L5430, plenty of disk and 1Gb RAM, HPUX 11i.

The machine has happily run for 2 or 3 years with no need for tuning / monitoring. So we no longer really have the expert knowledge to do this, and of course now we're getting performance problems.

We run Informix Dynamic Server V 7.31 and have recently migrated from the 32 to the 64 bit ports. Our problems seems to be worse since then. However we have also recently migrated our LAN infrastructure from NT to AD.

We're seeing two types of problem.

1. high load on the server which results in poor response times for users.

2. TCP errors and screen corruption or syncing problems in the client apps, which are simple terminal emulators.

Ok, now, for the first one I would like to use a tool like top, but one that logs the data so that I can look back over time and see what processes were running and what resources they were taking. Basically something that points a finger at a particular application or process, i.e. a user running a big report.

For the second one, we're stumped, Glance shows no network problems, we have nothing in the syslog on the server, not sure if there are others we should be checking. This may not be related to the server at all, it could be network problems, exacerbated by the AD upgrade maybe? Our network guys are going to run some monitoring tools on the section of the LAN that the server is connected to tomorrow hopefully, so this may throw something up.

Can anyone offer us some advice or a direction to go in please?

Thanks in advance.

[Raj D.]

Hi Tony ,

Here is something you can go with and finidng something for performance bottleneck ,

# sar -u -M 5 5 [ Checking all cpu and their loads ]
# sar -v 5 5 [ Check system table ]
# glance -t [ Check system table ]

# vmstat 5 5 [ Check swap and paging activity ]
Check for pi/po values any nonzero value is indicating heavy swap usuage.

# iostat 5 5 [ grep for the disk u are suspecting loaded heavily . ]

# glance [ Phy Memory, swap etc. check ]

# top [ Check for cpu utilisation and top process ]

# netstat -n [ check for established connection]

# check syslog.log for any error.

# also check kernel parameter are specified correctly.

Hope this will help ,

Enjoy ,
hth,
Raj.

[Tony Flaherty]

Raj,

Thanks for the reply. We have tried most of these with inconclusive results. A colleague will post more detailed results tomorrow.

Thanks again.

[Duncan Edmonstone]

Do you just have Glance, or in fact do you have GlancePlus Pak?

Many people have Glanceplus Pak and don't realise that they already have software which is recording historical data for them. The product is called either Mesureware or Openview Performance Agent (depending on whether you talk to the techies or marketing!).

Anyway you can check if you already have Mesureware by typing:

mwa status

if this comes back with some meaningful info then you do have measureware.

If you don't, you can always follow the links to download a trial copy from here:

http://www.openview.hp.com/products/ovperf/index.html
(you might have to apply for a HP passport, but thats straighforward enough)

You can get all the data you want out of Measureware using the 'extract' command, although it can be a bit complex! An easier route might be to download the evaluation copy of Performance Manager (aka PerfView) also available at the link I suggested.

With perfview its very easy to 'drill down' to actual historical process data - exactly what you want.

HTH

Duncan

[Geoff Wild]

With Perfview, I do this - and it mails me stats every night:

From root's cron

# performance
55 23 * * 1-5 /opt/perf/bin/extract -xp -r /home/gwild/zgbltemplate -g -b today 7:00 -e today 18:00 -f stdout | /bin/mailx -s 'svr1000 performance report' gwild >/dev/null 2>&1


# cat /home/gwild/zgbltemplate
REPORT "MWA Export on !SYSTEM_ID"
FORMAT ASCII
HEADINGS ON
SEPARATOR="|"
SUMMARY=60
MISSING=0
DATA TYPE GLOBAL
YEAR
DATE
TIME
GBL_CPU_TOTAL_UTIL
GBL_MEM_UTIL
GBL_SWAP_SPACE_UTIL
GBL_MEM_CACHE_HIT_PCT
GBL_DISK_UTIL_PEAK
GBL_FS_SPACE_UTIL_PEAK
GBL_MEM_PAGEOUT_RATE
GBL_MEM_SWAPOUT_RATE
GBL_MEM_QUEUE

Rgds...Geoff

[Raj D.]

Hi Tony ,

If you have MeasureWare Software/UX installed , the above seems to be a very good script.

And also the primary tools (sar , top , vmstat , iostat, netstat, glance) need to be use to check system performance. Also you should know the earlier system performance and system behaviour , to find it out better way. And also need to keep attention to the system , for over a period of time. You may write script to collect data everyday or hourly basis.

Enjoy ,
hth,

Raj.

[Bill Hassell]

32 to 64 bits isn't always a simple task. Your exectuables are larger and for better performance, Informix may need a *lot* more RAM for each instance of a database. 1Gb for Informix 7.31 and a single database is probably OK, but you may need to double or triple your RAM to see a significant improvement in performance (along with Informix config changes). You can also check vmstat to see what the po column says. If po is 2 digits (ie, 10) or more when the system is busy, you are severely short of RAM.

As far as AD versus NT, this isn't a LAN protocol, it is a PC administration tool and unless your LAN is severely overloaded, the AD switch may not be the issue. Look at lanadmin for your network stats. Assuming you are using 100Mbit full-duplex connection to your switch, there will be no FCS errors and no collisions. If there are, there is a duplex mismatch which will cause lots of errors (lost packets).

[Mike Peters_2]

Here are some more details as Tony promised. Running sar periodically shows swpot/s as between 0.80 and 1.00 under normal load but spiking periodically to 2 and above. We are having particular difficulty pinning down what process exactly is causing these periods of heavy paging. I've got mwa running now so we'll see what that tells us - are there any particular configuration options which will help us see which particular processes are paging heavily during these spikes?

As for the network traffic, I've used IPTraf under Linux but am unaware of a similar alternative for HP-UX. Also ntop, but my experience of that, again under Linux, is that it uses a lot of system resources - does anyone know of a similar, less resource hungry alternative?

Thanks for your help so far.

[Steven E. Protter]

Shalom Tony,

There are a number of kernel parametrs that are 64 bit specific that may need adjustment to handle 64 bit applications.

kmtune | grep 64

Then look at the settings for the 32 bit counterparts.

A lot of folks running older 32 bit apps never tuned the 64 bit parameters because they didn't have to. I got hammered by that policy at JUF when we migrated to 64 bit Oracle. Suddenly I had some tuning to do and it got worse as the Oracle Applications slowly migrated to 64 bit.

In the end our solution was to buy memory and tune the system and that made the problem go away.

http://www.hpux.ws/system.perf.sh

A decent set of tools, improvements on something HP gave me to deal with performance issues in the distant path.

Good Luck and let us know what happens.

SEP

[Mike Peters_2]

SEP

We've already configured maxtsiz_64bit and maxdsiz_64bit as 1Gb and maxssiz_64bit as 79MB
which I guess should be enough, according to the HP Tuning and Performance Guide.

Thanks for the input though.

[Bill Hassell]

You can find the resident size of all the processes sorted largest to smallest with this command:

UNIX95=1 ps -e -o vsz,pid,ruser,args | sort -rn | head

This shows the largest processes. If vmstat shows po more than 1 digit, the largest programs will be shown at the top of the list. Use swapinfo -tam to show actual swap usage. If the po value and swapinfo report swap space is in use (as opposed to reserved) then you need more RAM, or revonfigure your Informix database engine to use less RAM (which will probably slow down the database too). There is no substitute for RAM when performance is a problem.

[Ted Buis]

I agree with Bill that from your description that more RAM would be needed, but you should also check for patches, look at how full the file system is and check out your network. TCP errors don't come from any directory server unless there is bad hardware. Try ping to test network speed and errors if you don't have something better. HP-UX 11i barely gets going at 1GB. I never recommend less than 2GB for it. HP-UX 11.0 could get by with less. Use Glance to see if you are doing page outs, and if so get more RAM. You may also need to increase swap space with larger executables. Post "swapinfo -tam".