System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Need tuning advice for server

SOLVED
Go to solution
S. Ney
Trusted Contributor

Need tuning advice for server

Hello,

I am seeking some tuning advice from some of the forum guru's. I have an rx6600 running 11.31 March09. I would like to update 11.31 with newer patches however with our moratoriums and change culture it is not an immediate option.

The server has 32Gb of RAM and is attached to an XP24000 array. The server is running oracle version 11.1 64 bit (Non-RAC, Non-ASM). It also runs Goldengate which populates the database. It has a very large SGA (upon advice from oracle and our dba's which can be seen in the ipcs commands.)

I've had cases open with HP in the past and have received tuning advice and we have adjusted params several times as well as patch when able. I also have quite a few tuning documents. However the demands on the server are increasingly greater as more applications are tied into the database and pull data via odbc jobs. We're working on bringing up a second server in order to split the load but I'm looking for some tweaks that will relieve some of the disk and memory pressure.

The volume group layout is striped pvg's of 3x100 for the database lvols. Mirrored for the local disks. The least busy disks belong to a large backup lvol. I can't really distribute the load to other disks as they are all around 100%

Two variables I think would help would be to increase max_q_depth via scsimgr from 8 to 16. I've done this on disks 314-316. I also think tweaking filecache_max and min may help. However I haven't had much experience with in-depth tuning.
I've attached sar, vmstat, swapinfo, iostat and kctune.
10 REPLIES
Steven E. Protter
Exalted Contributor

Re: Need tuning advice for server

Shalom

reducing filecache_max and min may alleviate memory issues.

Here is what strikes me:

10:45:30 disk6 20.90 19.78 0 47 758 67.85 21.99
disk7 21.39 20.69 0 47 742 81.38 24.96
disk9 51.24 31.27 0 104 1622 148.35 37.07
disk15 40.80 30.53 0 102 1596 96.77 22.42


10:45:44 disk6 0.50 0.50 1 0 16 0.00 5.09
disk9 1.49 0.50 0 2 15 0.00 8.28
disk15 1.49 0.50 0 2 15 0.00 9.34
disk293 100.00 0.50 130 129 5039 0.00 7.77
disk294 100.00 0.50 130 136 4370 0.00 7.94
disk295 100.00 0.50 137 137 4712 0.00 8.02
disk296 96.52 0.53 77 27 16892 0.05 12.14
disk297 100.00 0.51 106 28 18937 0.02 9.80
disk298 94.03 0.50 72 25 14034 0.00 11.47
disk299 5.97 0.50 3 10 84 0.00 4.72
disk300 3.98 0.50 1 4 259 0.00 7.82
disk301 9.45 0.50 3 9 537 0.00 7.62
disk302 100.00 0.50 137 103 22205 0.01 9.23
disk303 100.00 0.50 131 82 25974 0.06 11.01
disk304 100.00 0.51 150 83 25775 0.08 9.06
disk305 5.47 0.50 0 11 3156 0.00 4.98
disk314 100.00 0.50 85 18 16000 0.00 13.66
disk315 85.07 0.50 73 18 12832 0.00 12.70
disk316 100.00 0.50 75 17 14718 0.00 13.47

The area I would focus on is disk i/o.

If these disks are provided by SAN you might want to see if they can be tweaked to improve disk i/o.

You may want to work with the dba's and see if the disk can be laid out in a different way that provides better i/o throughput or avoids hot spots.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
S. Ney
Trusted Contributor

Re: Need tuning advice for server

Steven,

Thanks for your response.
I was thinking of changing filecache_max and min to 8% and 5% respectively. I wasn't sure the impact of this change however.

Disks 9&15 belong to vg00, disks 6&7 vgora. Those disks are the internal SAS disks. Disks 290& up are on the array. The disks with the least amount of load belong to archive and backup lvols so I can't really extend the existing lvols to those disks. The dba's won't move any tables or indexes.

Here's a pvg breakdown.
VG /dev/vginfoddp
PVG vginfoddp_pvg01xp
/dev/disk/disk293
/dev/disk/disk294
/dev/disk/disk295
PVG vginfoddp_pvg02xp
/dev/disk/disk296
/dev/disk/disk297
/dev/disk/disk298
PVG vginfoddp_pvg03xp
/dev/disk/disk299
/dev/disk/disk300
/dev/disk/disk301
PVG vginfoddp_pvg04xp
/dev/disk/disk302
/dev/disk/disk303
/dev/disk/disk304
PVG vginfoddp_pvg05xp
/dev/disk/disk305
/dev/disk/disk306
/dev/disk/disk307
PVG vginfoddp_pvg06xp
/dev/disk/disk308
/dev/disk/disk309
/dev/disk/disk310
PVG vginfoddp_pvg07xp
/dev/disk/disk311
/dev/disk/disk312
/dev/disk/disk313
PVG vginfoddp_pvg08xp
/dev/disk/disk314
/dev/disk/disk315
/dev/disk/disk316
PVG vginfoddp_pvg09xp
/dev/disk/disk317
/dev/disk/disk318
/dev/disk/disk319
PVG vginfoddp_pvgbackupxp
/dev/disk/disk320
/dev/disk/disk321
/dev/disk/disk322
/dev/disk/disk323
/dev/disk/disk324
/dev/disk/disk325
/dev/disk/disk326
/dev/disk/disk327
/dev/disk/disk334
/dev/disk/disk335
Tim Nelson
Honored Contributor

Re: Need tuning advice for server

lvm striping of disk array luns in some cases may defeat the disk array caching and impede performance. in other cases it helps.

test both ways and determine for yourself.

S. Ney
Trusted Contributor

Re: Need tuning advice for server

This is the second build of this server. We did not originally have pvg's. The striping across disks has offered better performance as the data is spread across more disks and has helped alleviate some of the original hotspots. We've tried to spread the busiest lvols across disks of their own.

I am hoping for some more suggestions as to whether or not there are additional kernel parameters that I should look into tuning. Any changes made I have to state why changing them would be beneficial to my mgmt.

thanks,
Rita C Workman
Honored Contributor
Solution

Re: Need tuning advice for server

Here's my 2 cents...

Reduce your filecache max/min. I'd probably go with 10/5.
If you run sar -v you'll find that your ninode is way too high. I'd reduce that down more to what you sar output shows you.
In conjunction with ninode, I also always change vx_ninode. For a 32Gb box, I'd put that at around 100,000 to 200,000. Sounds big, but leaving it at "0" is actually allowing the O/S to make it bigger than those settings.
Both of these parms create large tablespaces that aren't needed. When you look at your sar -v you will see how many inodes you are using/what your parm is set at. Look how big a difference. These are cases, where making the parm smaller gives you more.

Now, I know Oracle always wants some ridiculous SGA and wants maxdsiz set high. Not sure about your shop or your apps, but on Oracle 10 I'd set it up for them to load the software and I would put it back down after. In other words to about half or a third of what you have it set at. But again..you'll have to decide on that.

Didn't look down all of the parms - just hit a few favorites of mine.

Rgrds,
Rita
Rita
Emil Velez
Honored Contributor

Re: Need tuning advice for server

filecache max and filecache min should be minimal at 1 and say 5 %

make sure the shmmax is set to larger than the size of your largest SGA segment.

SGA is implemented as shared memory so you will see it in IPCS

when you change the scsi queue make sure it is helping by looking at the queue length and average service time in glance or sar -d
S. Ney
Trusted Contributor

Re: Need tuning advice for server

I'll have to schedule a request to change filecace_max and min even though they are dynamic. But I appreciate all the suggestions about altering that kernel param.

Rita,

thanks for your suggestions, I haven't thought about ninode or vx_ninode. I think what you stated is sound advice and I'll look into changing those in a future maintenance.

The SGA is very large for this database, (12Gb) and shmmax is set to system memory. The dba's feel that sga size is appropriate.
Rita C Workman
Honored Contributor

Re: Need tuning advice for server

DBA's always do....
S. Ney
Trusted Contributor

Re: Need tuning advice for server

Have some good suggestions, I will implement and test over time.
chris huys_4
Honored Contributor

Re: Need tuning advice for server

Hi,

I think actually you should increase the sga size. From the performance statistics at this moment the system has multiple gigabyte of free memory (8?), which could be used by (an increased) database cache to reduce the amount of disk IO that is being done. (if the disk performance statistics are being performed at the moment the system was doing database "things"). And as disk IO looks to be the primary bottleneck, every help is probably welcome.

Also first thing, I would do, is to remove the software striping. Software striping only helps on storage that are specifically designed to "help" "the storage striping to perform". If the storage group doesnt know, the system admin group is using storage striping, then probably the software striping will degrade performance instead of increase performance, and in that case its better to remove the software striping.

NOTE: In the above case, I would probably not increase the max_q_depth parameter, avwait is not showing any IOs that are waiting on the host, and the avserv is allready quite high (>10msec) with the default max_q_depth. Increasing the max_q_depth may therefor, degrade performance, instead of increasing it..

NOTE2: Its a bit strange that there are multiple luns that show 100% at the same time. Could it be due to the software striping, that all this luns are hitting the
same disks on the xp24k and therefor are "interconnected" ?

Greetz,
Chris