topic Re: Setting kernel params after Mem and CPU upgrade? in Operating System - HP-UX

Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 02:37:44 GMT

Hi!
We got a machine with HP 11.23 upgraded with more memory and additional CPU. Although, the performance of all progs decreased over 15%. This is strange, as the server has more memory and CPU now. I was researching about any kernel parameters that must be updated, but I am not very experienced with kernel tuning. Can you imagine what could be wrong?
Thank you!

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 03:00:35 GMT

I can imagine plenty of things, but couldn't really comment on any of them with the level of data you have provided!

Questions:

What model of HP server is this? How many CPUs did it have before and after, how much memory before an after?

What is the application, how are you determining that it is performing 15% slower?

How did you reach the conclusion that more memory/CPUs is what you needed to speed up your application?

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 03:17:52 GMT

Hi!
This is an rx4640 with HP-UX 11.23
It has 4 CPU's and 32 GB memory. It is running primarely an SAP-BW instances. But I also noticed that perl-data collectors, that I wrote became slower. We needed more memory, because the SAP was eating it all (16GB) and the system became unstable (process deactivating and swaping too much). Now it works fine, but slower. For a db-load we needed 13hrs before upgrade and for the same amount of data we need more than 16 hrs now.

Re: Setting kernel params after Mem and CPU upgrade?

Steven E. Protter — Mon, 23 Apr 2007 03:37:09 GMT

Shalom,

Shalom, remember I/O is the primary bottleneck on most systems these days. How fast disk writes are happening may have nothing to do with CPU and memory. It may have to do with the drive speed. Some Itanium systems have 10K SCSI instead of 15 K SCSI that was the standard in PA-RISC systems. This slows things down.

This doc will help you with tuning for performance.

http://www1.itrc.hp.com/service/cki/search.do?searchString=UPERFKBAN00000726&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchCrit=allwords&search.x=28&admit=-682735245+1177317422000+28353475&category=c0&mode=id&search.y=8

SEP

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 04:14:37 GMT

Yes, I would share SEP's initial asessment - if adding more CPUs slows things down that almost suggests more streams of work are being pushed at the disks...

If you've added memory and some of that is free you should belooking to increase the size of the Oracle SGA, and also keep a check on the size of your buffer cache. What size do you have dbc_min_pct and dbc_max_pct set to?

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 04:46:22 GMT

kctune said:

dbc_min_pct 5 Default Immed
dbc_max_pct 5 5 Immed

Why should more I/O appear after CPU and memory came, the applications are the same.

Thank you.

Re: Setting kernel params after Mem and CPU upgrade?

Rasheed Tamton — Mon, 23 Apr 2007 04:55:12 GMT

Most probable cause might be the buffer cache as said above.

Please give the outputs of
# sysdef|grep dbc

(
It wiil give the output of
dbc_max_pct
dbc_min_pct
)

and

#swapinfo -tam

You might get more expert opinions based on that.

SEP, I am nto able to see the link. I am getting always a "System unavailable" message.

Regards.

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 05:05:43 GMT

Thank you,

The output is in the attached txt file.

SEP: I also cannot see the link, it seems that the document was renamed or deleted.

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 05:20:44 GMT

Kalim,

It's not a great analogy, but think of your system as a water supply system.

Memory is the resevoir, CPU are the pumps, disk is the pipes out to consumers.

Now if the pipes were your bottleneck anyway, then adding more pumps and creating a bigger resevoir could actually make thinsg *worse*.

Like I said, not a good analogy! Do you have any disk IO metrics which might prove/disprove this theory? A simple sar -d 10 10 during the job thats taking lonegr might be a good starting point...

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 06:53:27 GMT

This is the output (last lines),

13:21:01 c2t0d0 1.60 0.50 3 26 0.00 8.76
c2t1d0 1.00 0.50 2 23 0.00 5.58
c4t0d0 87.20 0.62 620 32503 0.09 4.04
13:21:11 c2t0d0 1.90 0.50 3 34 0.00 11.69
c2t1d0 1.00 0.50 2 31 0.00 7.67
c4t0d0 88.51 0.62 551 29445 0.08 4.44
13:21:21 c2t0d0 0.90 0.50 2 23 0.00 8.57
c2t1d0 0.60 0.50 1 21 0.00 6.03
c4t0d0 91.00 0.58 526 28868 0.05 4.60
13:21:31 c2t0d0 2.20 0.50 4 36 0.00 7.82
c2t1d0 1.10 0.50 3 31 0.00 4.70
c4t0d0 88.89 0.59 505 28306 0.06 4.48
13:21:41 c2t0d0 1.30 0.50 5 68 0.00 3.54
c2t1d0 0.60 0.50 1 16 0.00 5.56
c4t0d0 88.61 0.69 423 24461 0.20 5.25

Average c2t0d0 3.14 2.73 7 305 7.52 12.78
Average c2t1d0 1.73 3.17 5 63 7.39 10.76
Average c4t0d0 88.86 41.38 558 28633 34.68 4.56

This is when the SAP-Data-Load-application is running. I see the bottleneck here.

I have old data collected, but only for memory and load, not for diskIO, but I will watch the system for some days to see what is happening.

Thank you for your answers.

Re: Setting kernel params after Mem and CPU upgrade?

Steve Lewis — Mon, 23 Apr 2007 07:12:22 GMT

When performance tuning it is always necessary to consider the whole system.
You added CPU and memory, but did you also tune the database and storage to go with it?
You appear to have a bottleneck on one disk.

As a general point, a batch process such as a data load will consume all the resource allocated, sometimes to the detriment of another part of the system.
Increasing the processing power may have incidentally caused an overload of your database writers, causing head thrashing and huge queues to build up on the disk.

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 08:14:18 GMT

Kalin,

I'm assuming that c2t0d0 and c2t1d0 are youre mirrored boot disks. They don't seem to be so busy.

What is c4t0d0 ? is is a LUN on a SAN or is it a local disk? What do you have on this disk? presumably a filesystem (pvdisplay -v /dev/dsk/c4t0d0 should show us what logical volumes if any you have on this disk).

None of these figures look particularly alarming though, so maybe we're chasing the wrong thing here. How does the SAP load work? i.e. where does it load from and to in terms of disks/filesystems?)

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Steven E. Protter — Mon, 23 Apr 2007 08:29:55 GMT

dbc_min_pct 5 Default Immed
dbc_max_pct 5 5 Immed

I found setting max to 7 works 11.11.

Bill Hassell has seen some evidence of buffer cache at higher levels helping Oracle performance on cooked filesystems and HP-UX 11.23

SEP

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 08:38:47 GMT

Hi Duncan,

you are right, c2t0d0 and c2t1d0 is the mirrored boot device and c4t0d0 is a disk array with a bunch of harddrives connected through fiber-channel cards (SAN).
The output is in the atached file.
In this case the SAP-System is reading and writing data from the SAN to the SAN, there is stored the whole data, since the DB is there. Therefore the great disk usage.

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 09:19:07 GMT

ok, next questions:

1) Do you have 2 or more paths to this disk or just one? Does your server just have the one FC card init it?

2) what type of disk array is it? Do you know how many and what type of disks actually provide the LUN? The thing is that in sequential IOs which we usually see during loads, reading and writing to the same disk(s) can be an issue, even if there is a disk array with cache involved.

3) If you only have one LUN, you should probably look at a fairly high scsi queue depth (number of IOs that can be sent concurrently to the disk array). Check its current value using:

scsictl -m queue_depth /dev/rdsk/c4t0d0

if its set to the default of 8 you can probably increase it - the maximum value will depend on your disk array, so consult with your SAN admin or array vendor.

Of course all of this assumes that the problem is with IO, which we can't really be sure of yet. You said you looked at sar outputs both before and after the memory and CPU were added. What did you see?

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 09:39:33 GMT

There is only one path.
Server has two cards.
Disk Array is MSA30 14x73Gb 15k rpm
queue depth is 8.

Unfortunately I don't have any I/O-data before the upgrade. I have only load and memory, but not disk. Both load and memory usage decreased after upgrade..
Thank you for your support.

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 10:14:30 GMT

OK, so the MSA30 is a SCSI JBOD disk tray - you must have some sort of disk array controller in front of that - either MSA1000 or MSA1500 if you really are attached by fibre, otherwise if you are attached via SCSI you might have an internal SmartArray controller.

What does

diskinfo /dev/rdsk/c4t0d0

state?

HTH

Duncan

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 10:43:39 GMT

SCSI describe of /dev/rdsk/c4t0d0:
vendor: HP
product id: LOGICAL VOLUME
type: direct access
size: 284503117 Kbytes
bytes per sector: 512

:(

Re: Setting kernel params after Mem and CPU upgrade?

Kalin Evtimov — Mon, 23 Apr 2007 10:46:56 GMT

The guy that manages the machine told me that it is connected via SCSI-Kabel, it is still not attached to the SAN

Re: Setting kernel params after Mem and CPU upgrade?

Duncan Edmonstone — Mon, 23 Apr 2007 11:01:16 GMT

Kalin,

so you have a SmartArray card in your system. We should be able to interrogate that to determine the configuration of your storage:

ioscan -kfnd ciss

Should find us the device file for the SmartArray controller.

saconfig /dev/cissX

replacing cissX with what you get back from the ioscan, should show us how the LUN is presented to the host.

HTH

Duncan