Setting kernel params after Mem and CPU upgrade?

[Kalin Evtimov]

Hi!
We got a machine with HP 11.23 upgraded with more memory and additional CPU. Although, the performance of all progs decreased over 15%. This is strange, as the server has more memory and CPU now. I was researching about any kernel parameters that must be updated, but I am not very experienced with kernel tuning. Can you imagine what could be wrong?
Thank you!

[Duncan Edmonstone · Accepted solution]

I can imagine plenty of things, but couldn't really comment on any of them with the level of data you have provided!

Questions:

What model of HP server is this? How many CPUs did it have before and after, how much memory before an after?

What is the application, how are you determining that it is performing 15% slower?

How did you reach the conclusion that more memory/CPUs is what you needed to speed up your application?

HTH

Duncan

[Kalin Evtimov]

Hi!
This is an rx4640 with HP-UX 11.23
It has 4 CPU's and 32 GB memory. It is running primarely an SAP-BW instances. But I also noticed that perl-data collectors, that I wrote became slower. We needed more memory, because the SAP was eating it all (16GB) and the system became unstable (process deactivating and swaping too much). Now it works fine, but slower. For a db-load we needed 13hrs before upgrade and for the same amount of data we need more than 16 hrs now.

[Steven E. Protter]

Shalom,

Shalom, remember I/O is the primary bottleneck on most systems these days. How fast disk writes are happening may have nothing to do with CPU and memory. It may have to do with the drive speed. Some Itanium systems have 10K SCSI instead of 15 K SCSI that was the standard in PA-RISC systems. This slows things down.

This doc will help you with tuning for performance.

http://www1.itrc.hp.com/service/cki/search.do?searchString=UPERFKBAN00000726&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchCrit=allwords&search.x=28&admit=-682735245+1177317422000+28353475&category=c0&mode=id&search.y=8

SEP

[Duncan Edmonstone]

Yes, I would share SEP's initial asessment - if adding more CPUs slows things down that almost suggests more streams of work are being pushed at the disks...

If you've added memory and some of that is free you should belooking to increase the size of the Oracle SGA, and also keep a check on the size of your buffer cache. What size do you have dbc_min_pct and dbc_max_pct set to?

HTH

Duncan

[Kalin Evtimov]

kctune said:

dbc_min_pct 5 Default Immed
dbc_max_pct 5 5 Immed

Why should more I/O appear after CPU and memory came, the applications are the same.

Thank you.

[Rasheed Tamton]

Most probable cause might be the buffer cache as said above.

Please give the outputs of
# sysdef|grep dbc

(
It wiil give the output of
dbc_max_pct
dbc_min_pct
)

and

#swapinfo -tam

You might get more expert opinions based on that.

SEP, I am nto able to see the link. I am getting always a "System unavailable" message.

Regards.

[Kalin Evtimov]

Thank you,

The output is in the attached txt file.

SEP: I also cannot see the link, it seems that the document was renamed or deleted.

[Duncan Edmonstone]

Kalim,

It's not a great analogy, but think of your system as a water supply system.

Memory is the resevoir, CPU are the pumps, disk is the pipes out to consumers.

Now if the pipes were your bottleneck anyway, then adding more pumps and creating a bigger resevoir could actually make thinsg *worse*.

Like I said, not a good analogy! Do you have any disk IO metrics which might prove/disprove this theory? A simple sar -d 10 10 during the job thats taking lonegr might be a good starting point...

HTH

Duncan

[Kalin Evtimov]

This is the output (last lines),

13:21:01 c2t0d0 1.60 0.50 3 26 0.00 8.76
c2t1d0 1.00 0.50 2 23 0.00 5.58
c4t0d0 87.20 0.62 620 32503 0.09 4.04
13:21:11 c2t0d0 1.90 0.50 3 34 0.00 11.69
c2t1d0 1.00 0.50 2 31 0.00 7.67
c4t0d0 88.51 0.62 551 29445 0.08 4.44
13:21:21 c2t0d0 0.90 0.50 2 23 0.00 8.57
c2t1d0 0.60 0.50 1 21 0.00 6.03
c4t0d0 91.00 0.58 526 28868 0.05 4.60
13:21:31 c2t0d0 2.20 0.50 4 36 0.00 7.82
c2t1d0 1.10 0.50 3 31 0.00 4.70
c4t0d0 88.89 0.59 505 28306 0.06 4.48
13:21:41 c2t0d0 1.30 0.50 5 68 0.00 3.54
c2t1d0 0.60 0.50 1 16 0.00 5.56
c4t0d0 88.61 0.69 423 24461 0.20 5.25

Average c2t0d0 3.14 2.73 7 305 7.52 12.78
Average c2t1d0 1.73 3.17 5 63 7.39 10.76
Average c4t0d0 88.86 41.38 558 28633 34.68 4.56

This is when the SAP-Data-Load-application is running. I see the bottleneck here.

I have old data collected, but only for memory and load, not for diskIO, but I will watch the system for some days to see what is happening.

Thank you for your answers.

[Steve Lewis]

When performance tuning it is always necessary to consider the whole system.
You added CPU and memory, but did you also tune the database and storage to go with it?
You appear to have a bottleneck on one disk.

As a general point, a batch process such as a data load will consume all the resource allocated, sometimes to the detriment of another part of the system.
Increasing the processing power may have incidentally caused an overload of your database writers, causing head thrashing and huge queues to build up on the disk.

[Duncan Edmonstone]

Kalin,

I'm assuming that c2t0d0 and c2t1d0 are youre mirrored boot disks. They don't seem to be so busy.

What is c4t0d0 ? is is a LUN on a SAN or is it a local disk? What do you have on this disk? presumably a filesystem (pvdisplay -v /dev/dsk/c4t0d0 should show us what logical volumes if any you have on this disk).

None of these figures look particularly alarming though, so maybe we're chasing the wrong thing here. How does the SAP load work? i.e. where does it load from and to in terms of disks/filesystems?)

HTH

Duncan

[Steven E. Protter]

dbc_min_pct 5 Default Immed
dbc_max_pct 5 5 Immed

I found setting max to 7 works 11.11.

Bill Hassell has seen some evidence of buffer cache at higher levels helping Oracle performance on cooked filesystems and HP-UX 11.23

SEP

[Kalin Evtimov]

Hi Duncan,

you are right, c2t0d0 and c2t1d0 is the mirrored boot device and c4t0d0 is a disk array with a bunch of harddrives connected through fiber-channel cards (SAN).
The output is in the atached file.
In this case the SAP-System is reading and writing data from the SAN to the SAN, there is stored the whole data, since the DB is there. Therefore the great disk usage.

[Duncan Edmonstone]

ok, next questions:

1) Do you have 2 or more paths to this disk or just one? Does your server just have the one FC card init it?

2) what type of disk array is it? Do you know how many and what type of disks actually provide the LUN? The thing is that in sequential IOs which we usually see during loads, reading and writing to the same disk(s) can be an issue, even if there is a disk array with cache involved.

3) If you only have one LUN, you should probably look at a fairly high scsi queue depth (number of IOs that can be sent concurrently to the disk array). Check its current value using:

scsictl -m queue_depth /dev/rdsk/c4t0d0

if its set to the default of 8 you can probably increase it - the maximum value will depend on your disk array, so consult with your SAN admin or array vendor.


Of course all of this assumes that the problem is with IO, which we can't really be sure of yet. You said you looked at sar outputs both before and after the memory and CPU were added. What did you see?

HTH

Duncan

[Kalin Evtimov]

There is only one path.
Server has two cards.
Disk Array is MSA30 14x73Gb 15k rpm
queue depth is 8.

Unfortunately I don't have any I/O-data before the upgrade. I have only load and memory, but not disk. Both load and memory usage decreased after upgrade..
Thank you for your support.

[Duncan Edmonstone]

OK, so the MSA30 is a SCSI JBOD disk tray - you must have some sort of disk array controller in front of that - either MSA1000 or MSA1500 if you really are attached by fibre, otherwise if you are attached via SCSI you might have an internal SmartArray controller.

What does

diskinfo /dev/rdsk/c4t0d0

state?

HTH

Duncan

[Kalin Evtimov]

SCSI describe of /dev/rdsk/c4t0d0:
vendor: HP
product id: LOGICAL VOLUME
type: direct access
size: 284503117 Kbytes
bytes per sector: 512

:(

[Kalin Evtimov]

The guy that manages the machine told me that it is connected via SCSI-Kabel, it is still not attached to the SAN

[Duncan Edmonstone]

Kalin,

so you have a SmartArray card in your system. We should be able to interrogate that to determine the configuration of your storage:

ioscan -kfnd ciss

Should find us the device file for the SmartArray controller.

saconfig /dev/cissX

replacing cissX with what you get back from the ioscan, should show us how the LUN is presented to the host.

HTH

Duncan

[Kalin Evtimov]

wow, great. A bit too much hardware for me.
The output is in the attachment.

Thank you.

Regards,

Kalin

[Duncan Edmonstone]

Looking at this output I can see that the LUN c4t0d0 is made up of 8 x 72GB disks in a RAID1+0 configuration to provide a 288GB LUN. I have to say that doesn't seem like much for a SAP BW environment to me, but I guess that depends on what you are doing with it... certainly I would expect the number of CPUs you have in the system now could more that max-out a single Smart Array card for IO.

Also as I pointed out earlier, all your reads and write are going to these 8 disks, which seems to me to probably be pretty inefficient.

Looking at the output there appears to be another drive configured (probably presented as c4t0d1) - what is that used for? I think you need to look at doing the SAP load so that it is reading from one set of disks and writing to another.

You could try upping the scsi queue depth, although I can't find a reccomended setting for the Smart Array card. It's probably safest to up it in small increments. If you see performance gains then keep going until you see no further perf gains, or start getting device queue full messages in the syslog.

I'd start with just 16 and see how that goes:

scsictl -m queue_depth=16 /dev/rdsk/c4t0d0

I'd be surprised if this makes much difference.

HTH

Duncan