Need tuning advice for server

[S. Ney]

Hello,

I am seeking some tuning advice from some of the forum guru's. I have an rx6600 running 11.31 March09. I would like to update 11.31 with newer patches however with our moratoriums and change culture it is not an immediate option.

The server has 32Gb of RAM and is attached to an XP24000 array. The server is running oracle version 11.1 64 bit (Non-RAC, Non-ASM). It also runs Goldengate which populates the database. It has a very large SGA (upon advice from oracle and our dba's which can be seen in the ipcs commands.)

I've had cases open with HP in the past and have received tuning advice and we have adjusted params several times as well as patch when able. I also have quite a few tuning documents. However the demands on the server are increasingly greater as more applications are tied into the database and pull data via odbc jobs. We're working on bringing up a second server in order to split the load but I'm looking for some tweaks that will relieve some of the disk and memory pressure.

The volume group layout is striped pvg's of 3x100 for the database lvols. Mirrored for the local disks. The least busy disks belong to a large backup lvol. I can't really distribute the load to other disks as they are all around 100%

Two variables I think would help would be to increase max_q_depth via scsimgr from 8 to 16. I've done this on disks 314-316. I also think tweaking filecache_max and min may help. However I haven't had much experience with in-depth tuning.
I've attached sar, vmstat, swapinfo, iostat and kctune.

[Steven E. Protter]

Shalom

reducing filecache_max and min may alleviate memory issues.

Here is what strikes me:

10:45:30 disk6 20.90 19.78 0 47 758 67.85 21.99
disk7 21.39 20.69 0 47 742 81.38 24.96
disk9 51.24 31.27 0 104 1622 148.35 37.07
disk15 40.80 30.53 0 102 1596 96.77 22.42


10:45:44 disk6 0.50 0.50 1 0 16 0.00 5.09
disk9 1.49 0.50 0 2 15 0.00 8.28
disk15 1.49 0.50 0 2 15 0.00 9.34
disk293 100.00 0.50 130 129 5039 0.00 7.77
disk294 100.00 0.50 130 136 4370 0.00 7.94
disk295 100.00 0.50 137 137 4712 0.00 8.02
disk296 96.52 0.53 77 27 16892 0.05 12.14
disk297 100.00 0.51 106 28 18937 0.02 9.80
disk298 94.03 0.50 72 25 14034 0.00 11.47
disk299 5.97 0.50 3 10 84 0.00 4.72
disk300 3.98 0.50 1 4 259 0.00 7.82
disk301 9.45 0.50 3 9 537 0.00 7.62
disk302 100.00 0.50 137 103 22205 0.01 9.23
disk303 100.00 0.50 131 82 25974 0.06 11.01
disk304 100.00 0.51 150 83 25775 0.08 9.06
disk305 5.47 0.50 0 11 3156 0.00 4.98
disk314 100.00 0.50 85 18 16000 0.00 13.66
disk315 85.07 0.50 73 18 12832 0.00 12.70
disk316 100.00 0.50 75 17 14718 0.00 13.47

The area I would focus on is disk i/o.

If these disks are provided by SAN you might want to see if they can be tweaked to improve disk i/o.

You may want to work with the dba's and see if the disk can be laid out in a different way that provides better i/o throughput or avoids hot spots.

SEP

[S. Ney]

Steven,

Thanks for your response.
I was thinking of changing filecache_max and min to 8% and 5% respectively. I wasn't sure the impact of this change however.

Disks 9&15 belong to vg00, disks 6&7 vgora. Those disks are the internal SAS disks. Disks 290& up are on the array. The disks with the least amount of load belong to archive and backup lvols so I can't really extend the existing lvols to those disks. The dba's won't move any tables or indexes.

Here's a pvg breakdown.
VG /dev/vginfoddp
PVG vginfoddp_pvg01xp
/dev/disk/disk293
/dev/disk/disk294
/dev/disk/disk295
PVG vginfoddp_pvg02xp
/dev/disk/disk296
/dev/disk/disk297
/dev/disk/disk298
PVG vginfoddp_pvg03xp
/dev/disk/disk299
/dev/disk/disk300
/dev/disk/disk301
PVG vginfoddp_pvg04xp
/dev/disk/disk302
/dev/disk/disk303
/dev/disk/disk304
PVG vginfoddp_pvg05xp
/dev/disk/disk305
/dev/disk/disk306
/dev/disk/disk307
PVG vginfoddp_pvg06xp
/dev/disk/disk308
/dev/disk/disk309
/dev/disk/disk310
PVG vginfoddp_pvg07xp
/dev/disk/disk311
/dev/disk/disk312
/dev/disk/disk313
PVG vginfoddp_pvg08xp
/dev/disk/disk314
/dev/disk/disk315
/dev/disk/disk316
PVG vginfoddp_pvg09xp
/dev/disk/disk317
/dev/disk/disk318
/dev/disk/disk319
PVG vginfoddp_pvgbackupxp
/dev/disk/disk320
/dev/disk/disk321
/dev/disk/disk322
/dev/disk/disk323
/dev/disk/disk324
/dev/disk/disk325
/dev/disk/disk326
/dev/disk/disk327
/dev/disk/disk334
/dev/disk/disk335

[Tim Nelson]

lvm striping of disk array luns in some cases may defeat the disk array caching and impede performance. in other cases it helps.

test both ways and determine for yourself.

[S. Ney]

This is the second build of this server. We did not originally have pvg's. The striping across disks has offered better performance as the data is spread across more disks and has helped alleviate some of the original hotspots. We've tried to spread the busiest lvols across disks of their own.

I am hoping for some more suggestions as to whether or not there are additional kernel parameters that I should look into tuning. Any changes made I have to state why changing them would be beneficial to my mgmt.

thanks,

[Rita C Workman · Accepted solution]

Here's my 2 cents...

Reduce your filecache max/min. I'd probably go with 10/5.
If you run sar -v you'll find that your ninode is way too high. I'd reduce that down more to what you sar output shows you.
In conjunction with ninode, I also always change vx_ninode. For a 32Gb box, I'd put that at around 100,000 to 200,000. Sounds big, but leaving it at "0" is actually allowing the O/S to make it bigger than those settings.
Both of these parms create large tablespaces that aren't needed. When you look at your sar -v you will see how many inodes you are using/what your parm is set at. Look how big a difference. These are cases, where making the parm smaller gives you more.

Now, I know Oracle always wants some ridiculous SGA and wants maxdsiz set high. Not sure about your shop or your apps, but on Oracle 10 I'd set it up for them to load the software and I would put it back down after. In other words to about half or a third of what you have it set at. But again..you'll have to decide on that.

Didn't look down all of the parms - just hit a few favorites of mine.

Rgrds,
Rita
Rita

[Emil Velez]

filecache max and filecache min should be minimal at 1 and say 5 %

make sure the shmmax is set to larger than the size of your largest SGA segment.

SGA is implemented as shared memory so you will see it in IPCS

when you change the scsi queue make sure it is helping by looking at the queue length and average service time in glance or sar -d

[S. Ney]

I'll have to schedule a request to change filecace_max and min even though they are dynamic. But I appreciate all the suggestions about altering that kernel param.

Rita,

thanks for your suggestions, I haven't thought about ninode or vx_ninode. I think what you stated is sound advice and I'll look into changing those in a future maintenance.

The SGA is very large for this database, (12Gb) and shmmax is set to system memory. The dba's feel that sga size is appropriate.

[Rita C Workman]

DBA's always do....

[S. Ney]

Have some good suggestions, I will implement and test over time.