Performance issue

[Helen French]

Hi,

I have 3 old HP servers in a site (2* K210 and G40). All the three servers are running the same applications (Oracle + Java + Sapphire + Netscape /iplanet ). The G40 is running with 11.0 (32bit) and two K210 are running with 10.20.
Now, when the development team tests an application upgrade on each server, the G40 works fine while the two K210 are running very slow. Both K210 are running with 2 processors (G40 with single CPU) and with more memory/swap than G40. The only difference is in the OS.

Here, does an OS upgrade to 11.0 (32bit) on the K class servers will make any difference ?

Does a kernel tuning will help me ?

what are the other things which I need to look in to ?

Thanks in advance for all your response !

Shiju

[James R. Ferguson]

Hi:

As a first approximation, I'd compare the kernel parameters of the two servers. I'd use SAM to view them since you don't have 'kmtune' at 10.20, and 'sysdef' may have some anomolies at 11.x

Regards!

...JRF...

[Tim D Fulford]

shiju

My gut instinct is that you have a memory bottle neck on the K's (as you said the G has more memory). However, this is rushing to a conclusion.

1 - get some statistics from sar, top etc or my favorite MeasureWare for all of the boxes.
so you could use the following to get disk & cpu utilisation for 20 seconds:
sar -du 20 1
virtual memory statistics:
vmstat
2 - I would start simple & check CPU usage, disk activity, swap etc then home in on the differences from there
3 - I would look at the difference between the kernel setings too, though 10.20 is different to 11.00 for some things.
4 - Are there any other H/W differences disk subsystem, network & so forth? Do you use the same type of filesystem Vxfs, HFS....

Without more info it would be difficult to point a specific problem.

Tim

[Helen French]

Exactly JRF,

that was my first step. I got it from SAM on all the three systems and now comparing those. Do I need to be more concerned about some specific parameters ? I think there will be some differences, the way 11.0 and 10.20 handles the kernel parameters (?)

Can I observe the kernel parameters overflow through Glance ? ( like sar -v)

Shiju

[Helen French]

Hi Tim,

1) You read it wrong. G40 has less memory. Both K210 has more memory and swap than G40.

2)There are H/W differences between systems. G40 uses all SE SCSI disks, where as K210 uses Model 20 diskarray and HASS boxes. So again, it should run faster.
3) The network speed remains the same.
4) G40 uses all vxfs file systems except /stand. K210 uses hfs mostly except some file systems with vxfs.

Shiju

[Tim D Fulford]

I realy must learn to read!

o HFS filesystems have variable (at set up time) block sizes. VxFS has 1kB fragment & 8kB block. If HFS filesystems are used for Oracle, do they have equivalent sizes?

o Do you have any performance statistics, sar, top, MeasureWare etc?

Tim

[Helen French]

Tim,

Yes I did check the performance through sar, top and glance. The user load is high in K210 when they test the software.

Shiju

[Tim D Fulford]

can you attach

sar -du 20 1

or a top output etc?

Tim

[James R. Ferguson]

Hi Shiju:

There is one favorite kernel parameter that a few of us have had experience with -- 'timeslice'. The usual database templates set this to <1> instead of the default <10>. Since this parameter controls the frequency of process context switching, setting it too low (genearlly less than 10) leads to doing process switching at the expense of real work.

Regards!

...JRF...

[Helen French]

Hi,

Tim - I will post the outputs shortly. I have to wait for the development team for testing their jobs again.

JRF - I checked the 'timeslice' parameter and it is '10 ' in all servers.

Any more inputs ?

Does anybody has any more suggestions or answers ?

[Mike Barron_3]

A couple things....


If the user % in CPU utilization is high and sys % and idle % is low, that is good. That means the application that the developers are running is using most of the CPU activity.

You said the G class has less memory, but did ya check your dbc_max kernel setting? default it is set to use 50% of your physical memory. We had a major memory bottleneck here and found out we should change that kernel setting to use 10%. It helped tremendously.

1 more thing just to add, check your disk utilization. Are you getting and disk bottlenecks?

[Helen French]

Hi Mike,

Thanks for the reply. I have %idle also high, but %sys comparatively low.

Tha dbc_max is default now in all servers, but i will change this as part of the Kernel tuning.

I will assign you points once I finished these tasks.

Thanks!
Shiju

[Sanjay_6]

Hi Shiju,

Take a look at the hp-ux performance and tuning guide,

http://docs.hp.com/hpux/onlinedocs/os/11.0/tuningwp.html

Hope this helps.

Regds

[Helen French]

Thanks Sanjay, ..that's an excellent guide !

Shiju

[Sridhar Bhaskarla]

Hi Shiju,

How about doing some kind of benchmarking on G40 and K Servers?. For ex., copy a 20 MB file from one file system to other and compare the difference. Make a C program that computes prime numbers, run it on both the systems and see the difference. Try ftping some stuff from the same source etc.,

I would certainly check to see if 11.0 helps. Borrow sometime from your development team. Make couple of make_recovery tapes on K server.If possible add an additional disk to K server and install 11.0 32bit on it. Import the file systems and ask your developers to run the application and see how it works. If not, you can always revert back to 10.20 either through make_recovery tape or the old boot disks.


-Sri

[Helen French]

Hi Sri,

I will defenitely check the options given by you and will assign points after that.

The 11.0 upgrade is my last option.

thanks for the rpely!
Shiju

[Helen French]

Hi Sri,

I did try the options u have listed and seems like the K-class server works fine than G.

I could see some difference in the kernel parameters and will try changing that, once i will be able to take a shutdown of the server.

Does anybody has any other suggestions / experiences ?

Shiju

[Sanjay_6]

Hi Shiju,

Here are two links which might be useful,

http://us-support.external.hp.com/cki/bin/doc.pl/sid=c70ed41c0e60c4e701/screen=ckiSearchResults?mode=id&searchString=KNC061098001

http://us-support.external.hp.com/cki/bin/doc.pl/sid=c70ed41c0e60c4e701/screen=ckiSearchResults?mode=id&searchString=A4460661

Hope this helps.

Regds

[Tim D Fulford]

Hi again

There is also a doc that I found useful which is S Ciullo & D. Grumann's "HP-UX Performance CookBook" he uses MeasureWare & Glance or GlancePlus which needs to be purchased if you do not already have it.

http://h21007.www2.hp.com/dspp/files/unprotected/devresource/Docs/TechPapers/UXPerfCookBook.pdf

Tim

[Ruediger Noack]

Hi,

have a look to %wio with sar -u.

Ruediger

[Helen French]

Hi,

Thanks for all your responses.

Does anybody know about the 'netisr_priority' ? I have it set to -1 in both K class systems and couldn't find it on the G class. Also netisr is running on both servers, but not in G. I checked the HP documents and found some details about these processes and priority, but beyond that if you have something more please share it.

Thanks!
Shiju

[Sanjay_6]

Hi Shiju,

Seems like the netisr_priority kernel parameter was there in 10.20. It has since been removed,

Here is a link from itrc for 10.20 netis_priority kernel parameter,

http://us-support.external.hp.com/cki/bin/doc.pl/sid=4e30c3ab1aae9b3056/screen=ckiDisplayDocument?docId=200000024625282

Hope this helps.

Regds
P.S. : what is your os version?

[Helen French]

Hi Sanjay,

The OS versions:

2 K210 systems - 10.20
G40 system - 11.0 (32bit)

Also, the link says '0' is the highest priority at which process handles networking packets. If I change the priority to '0', will that be ok ? any issues ?

Shiju

[Sanjay_6]

hi Shiju,

Just did a search on my system for netisr_priority in the kernel. the os version is 11.0. I found that the netis_priority is set to -1 on my system. I did a search on the patches for 11.0 for netisr and came up with two patches,

patch --- PHNE_23982
patch --- PHNE_25423

Hope this helps.

Regds

[Sanjay_6]

Hi Shiju,

The link also says

quote //

A value of -1 specifies that networking packets are handled on an interrupt basis. This yields the fastest possible packet handling rate.

//endquote

So i would suggest you should keep it at -1, until you have some real reasons to change it to "0".

Hope this helps.

Regds