Your experience on HP-UX system slowness..

[Patrick Ware_1]

Hello all,

I was asked by my manager to get a together a checklist of things to check for that would cause system slowness after an issue today. I can only come up with a few in my limited experience. I was curious to see what the experts had to say. If you so feel inclined, please indicate anything you could think of that you have experienced in your past. If there is a guide out there, feel free to point me in that direction too.

[Robert Salter · Accepted solution]

Hi Patrick,

Whenever we've experienced any system slow downs, I used glance or sar to try and isolate where or what the culprit(s) were. It could be memory useage/leaks or cpu utilization by a funky process, bad code, etc. Could I/O, bad disk(s), connection, etc. Or even a network connection, mismatched configs, bad cable, etc.

I know I put a lot of etc. in here but slowness could be attributed to any system resources whether under configured or improperly used.

Hope this helps.

cheers,

Robert

[Robert-Jan Goossens]

Hi Patrick,

--
experienced system slowness after an issue today
--

What was the issue, and how did you check the performance of your server? sar vmstat top...

Rgards,
Robert-Jan

[Steven E. Protter]

Shalom,

I would build a list based on reading some documents.

http://docs.hp.com/en/1219/tuningwp.html

http://h21007.www2.hp.com/portal/site/dspp/menuitem.476e81edc6fd24752c24b83f8973a801/?chid=74555e210a66a010VgnVCM100000275d6e10RCRD&topic=HP-UX&page=RTGL_HP-UX

First thing I would do is get some performance data with sar and then follow the data. I would not take a list and run it top to bottom.

http://www.hpux.ws/?p=6

SEP

[Pete Randall]

The most common causes are network, memory, disk contention, or cpu contention. With network, the main culprit is a duplex mismatch between port and switch - check with lanadmin. For memory, paging is the problem - check with "vmstat 1 5" and look at the po column - significant numbers here (double digits) mean paging is killing you. Disk contention and cpu contention are best examined with glance.


Pete

[Mark Ellzey]

Patrick,

One other thing to consider, related to I/O would be your database. If your performance issues are reflected in poor database performace, it could be bad indexes in the DB itself. Generally, this would show up constantly and you would have already fixed the problem. However, if the DB was asked to do something out-of-the-ordinary, and the indexes involved had not been optimized, you would experience massive degradation.

Ultimately, this would be reflected in high I/O activity.

Regards,
Mark

[Tim Nelson]

I am inclined to follow Pete's path. The following is a breakdown of the three main

Computing resources:
CPU, memory, disk ( and their relating configurations )

Network:
NIC, switch, routers, WAN, LAN, users interface.

Applications:
A huge pile of programming code running on one or possibly multiple systems.... :)

So the checklist.
-Is it computing resource issue ? review CPU, Mem, Disk
Is it a network/communication issue ? review LAN, WAN, remote LAN bandwidth or delays.
-Is it an application issue ? Due to the depth in nature to most applications start digging here once the above have been reviewed.

In some cases applications use resources poorly and tie directly into what is seen with CPU, Mem, and disk usage. A balanced approach typically is taken to provide additional faster resources, tune the resources appropriately for the workload or tune the application for more efficient usage.

Keep in mind that unless there is some hardware failure a computer only does what it is told to do.

[James R. Ferguson]

Hi Patrick:

To turn a phrase, "Got Glance?".

Seriously, another invaluable tool is the 'glance' performance monitor. If you don't have it licensed and installed, you should find an evaluation copy on one of the Application CDs.

Regards!

...JRF...

[Patrick Ware_1]

Yeah, I use Glance daily. A lot of the methods everyone describes, are already in practice by me. It is just hard for me to think of them unless my fingers are on the keyboard manipulating some server.

[TwoProc]

Patrick,

Unfortunately, I've found that you can write alerts all day long... but the fastest way I know of to tell when the system is slow, is that the phone IMMEDIATELY starts to ring: multiple lines.

As far was what makes a system slow, I think that this will vary by company. Over here, it's someone kicking off something large and untuned in Oracle, and in parallel no less, because it wasn't running well. Or, I've got a user locking up a master record and walking away to have a break, and the user blocking lock list just grows and grows.

If you were to start building a "tool" of sorts, I think a good start would be what made you run slow yesterday, and write a script to maybe detect it and mail it to your beeper/phone/whatever. Then add some of the common ones mentioned in the previous postings.

I've got some of these, but none of them beat multiple phone calls with each person giving me just a few words of problem description and where. From that experience, I almost always know where to look ... and then five minutes after I'm already working to resolve the issue I get an SMS message and a beep telling me what I already know...

It's good when I'm at lunch, and if the operator doesn't pick up the phone fast enough to call me, but that's really very rare. In other words, I almost always already know before I'm alerted... but not all of the time, so it's still good to have.

So, the next time it happens, and you've got your new cool "detect-o-matic" running, your next problem will be explaining to your boss how the phone rang with all of these people calling before your detect-o-matic let you know you had an issue... (actually happened JUST LIKE THAT over here at first) :-)

This is because when you've got problems it takes all of 5 seconds before the phone starts ringing, and the pass-through time for an email alert is usually expressed in minutes!

So, go ahead and work on it, you'll like it and it'll be cool. And you should have it, but the utility you gain from it probably won't be as high as what your boss thinks it should be. It will be more valuable to you when you're not there than when you are there.

[Patrick Ware_1]

I have a list compiled prelimenary list of the bullet points I was looking for:

* Kernel parameters

* System load (example: How many oracle and/or batch processes are running?)

* Disk contention on the host, and/or on external storage, Mis-Matched Power Path policy settings will cause high service and wait times.


* NFS Mount Issues (Did a mount become stale)

* Network configuration (are you using the correct DNS server, ?)

* Swap (Is there enough?)

* Memory (Is there enough, or are the semaphores exhausted?)

[Court Campbell]

Too add to your list:

* Is this the end of month?

[TwoProc]

Re: NFS Mount issues:

Your whole script (outermost check everything script) never finishes because the subscript to check NFS never finishes because the df on the stale mount point just hangs... :-(

Been there on this one...

That's why I've got another script that tells me by xxxx time, whether or not my other script is still out there, hung up trying to diagnose something that's hung. So, if by a certain time my check script is still running, I know it's hung and I get an email... sigh...

[Patrick Ware_1]

I am not trying to create a script, I am just getting together some info for a WIKI site with info, for those day when you've been up for 3 days on-call, and your brain doesn't work anymore...

[Bob E Campbell]

Probably too obvious to mention, but performance problems can be a symptom of hardware or security problems. Making sure *something* is looking at logs is important, but security should be scheduled before it is a problem.

[Rita C Workman]

Like the other folks I run down that list.

One last thing you can do is grab a good bit of packets on the process that's the high hitter. Looking down the output of something like "tusc" you (and your team) can spot something strange, or some bogus call that is slamming the box. Strange is rarely good, and for us...strange was our problem. I'm not experienced at reading these kind of things, but even I noticed the problem. It took a backline engineer to figure how to address it. But the first big key to fixing anything is always finding the problem.

So sometimes ... you gotta dig for it.

Rgrds,
Rita