Oracle/Threads/Number of Processors

[ronlevyca]

We are running two rp7420s in a clustered Oracle RAC configuration running HPUX 11.23. They each have four dual-core processors, so they appear to have 8 processors.

We are considering upgrading, and possible upgrade paths include a four processor quad-core x86_64 machine which would appear to have 16 processors, or maybe a two processor eight-core eight-thread UltraSPARC T2 based machine which would appear to have 128 processors.

So the first question is: Which is best for Oracle? And the answer is of course that it depends on what you are doing. The x86_64 machine will be 3.0 Ghz doing 16 things at once and the UltraSPARC T2 machine will be 1.4 Ghz doing 128 things at once, if there are 128 things to be done at all. (The Sun machine costs more physically and costs more Oracle licenses of course.)

So my question is, how can I determine how many threads I have active at any one time on my rp7420 running HPUX 11.23? Would it just be counting up how many processes are in the process table? Or is there a better measure to use?

[Steven E. Protter]

Shalom,

Check the kctune -l output there should be something more specific, perhaps even a formula.

check nkthreads or kctune -l | grep -i thread

I would think your best upgrade path if you want to continue using HP-UX and you should is to Itanium IA-64 systems. Those will probably out peform an x86_46 box running Linux (This box won't run HP-UX)

On the topic of threads:
http://unix.derkeiler.com/Newsgroups/comp.sys.hp.hpux/2005-12/msg00133.html

http://fixunix.com/hp-ux/143504-max-threads-allocating-limit-problem-hp-ux.html

http://unix.derkeiler.com/Newsgroups/comp.sys.hp.hpux/2005-12/msg00133.html

SEP

[ronlevyca]

My max_threads_proc (maximum number of threads per process) is 1024 and my nkthreads (maximum number of threads period) is 17079, which is a configured value based on our other Oracle-based settings I'm thinking.

But that doesn't tell me how many threads I have running right now.

Looking at GlancePlus, I see that I can get a Process List and a Thread List and that should tell me what I want to know.. it's just that there's too many in my graphical display window to count the lines easily. :) Thanks!

(And as for Itanium, we aren't necessarily tied to HPUX or Linux or Solaris for the future. Itanium has certain potent powers, but it's also costly compared to commodity x86_64 and we are hoping that for our small-read-heavy environment we won't be needing those powers.)

[Duncan Edmonstone]

A few points:

1) Oracle isn't currently a multi-threaded application (might not be true if you are running the MTS version of the listener - can't say I've looked hard at that implementation) -so in this case if nothing else but Oracle RAC is running on the host, simply count the processes... the OS itself shouldn't be doing anything significant even though no doubt some OS daemons are multi-threaded.

2) If you *insist* on moving away from HP-UX (and I wouldn't advise it...) then for a RAC environment I'd steer towards the x86_64 system for a couple of reasons - i) RAC's natural home is Linux - the clusterware simply has better hung node detection on Linux as it's able to insert code into the Linux kernel - something it can't do on HP-UX/Solaris for example (you get round it on HP-UX at least by using Serviceguard for cluster membership tasks - not sure about Solaris) and ii) I seem to recall that RAC can be sensitive to certain single threaded operations when under performance pressure - in this environment the greater number of slower threads on UltraSPARC might be an issue.

3) Please Please Please *don't* make the mistake of choosing you platform on acquisition cost alone (and I include cost of support in that equation). Acquisition costs are such a small part of the actual running of a system that they should be pretty much the least thing you consider when deciding what to purchase... unfortunately as they are the easiest to measure, people tend to focus on them to the exclusion of all other considerations. You should really conduct a TCO study on your options to determine the right approach - if you don't know how to go about this, talk to the vendors/resellers involved - if they want the business they should be able to help.

4) As this is a RAC configuration I'm assuming that availability is a key metric for this system? If that's the case then consider the resilience of any system you are going to replace it with... single system HA considerations should also be part of this even when you have a cluster.

5) I hope you're not comparing the list price of these systems together - you really won't get a realistic comparison off the back of that. Vendors operate differnt pricing models on different equipment, so make sure you tell all the vendors involved you want a discounted price and tell them you are looking at competitive options - you might find for example that a HP-UX option looks better price-wise once disocunts are applied...

HTH

Duncan

[ronlevyca]

Thanks very much. Yes, I'm aware of all these things and we are in fact going through exactly all the processes you describe.

Storage is the main driver for this, actually. Our backline storage is an XP512/XP48 combo, and it is unsupported as of the end of 2009. Our rp7420 is aging but still able to handle the load, yet getting another system would likely make for a much safer cutover than just another SAN.

In any case, I offer 10 points for anyone who can give me a command or script to get a linecount out of Glance (or some other method) for the number of processes total or the number of threads total. Glance's output is too wacky for the 'wc' program.

[Don Morris_1 · Accepted solution]

kcusage nproc
kcusage nkthread

?

[ronlevyca]

Ding! We have a winner!

[Duncan Edmonstone]

Whilst kcusage will give an interesting indication on the total number of processes and threads on the system, be aware that many of those processes and threads will not be doing anything (many system daemons fopr example will have many threads, most of which never do any work at all). I think this brings you back to using glance (as glance only logs data on "interesting" processes - i.e. those doing some work) - I have no chance to look at this today, but might be able to tomorrow...

HTH

Duncan

[Dennis Handly]

>for the number of processes total or the number of threads total.

You can use my pstat_getlwp(2) test source that I attached to this thread:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1296647

This output looks like:
inx: 590 lwpid 5448856 pid 53
inx: 614 lwpid 5445519 pid 53
inx: 662 lwpid 5449902 pid 53
inx: 683 lwpid 5448638 pid 53
inx: 692 lwpid 5448639 pid 53
inx: 703 lwpid 5448637 pid 53
...

$ ps -fp 53
root 53 0 0 Feb 8 ? 03:30:56 vxfsd

[Duncan Edmonstone]

Actually it looks like there is already an example glance advisor script which gives you close to what you are looking for.

Look at the example in the file thread_activity in the /opt/perf/examples/adviser directory

instruction on how to run this are included at the top of the file.

This prints some global stats and then prints out interesting threads based on a set of criteria... I simplified this down to a script which simply prints out threads with > 0.1% CPU activity... see attached.

To run just run 

glance -aos ./filename -j2

replacing ./filename with a path to the attached file

HTH

Duncan