Performance Issue?

[Geoff Wild]

John - great - that's what I thought.

BTW - as far as the OnlineJFS issue, I solved by:

restoring /etc/vx/elm directory from backup, then ran:

# /sbin/fs/vxfs/vxenablef -a

root@pc0031 [ /etc/vx/elm ]
# /sbin/vxlicense -p

vrts:vxlicense: INFO: Feature name: HP_OnlineJFS [50]
vrts:vxlicense: INFO: Number of licenses: 1 (non-floating)
vrts:vxlicense: INFO: Expiration date: Sun Jun 24 02:00:00 2007 (386.5 days from now)
vrts:vxlicense: INFO: Release Level: 22
vrts:vxlicense: INFO: Machine Class: All
vrts:vxlicense: INFO: Site ID: 0


# mount -o convosync=direct,mincache=direct /dev/vg01/lvol4 /zmnt


# mount |grep zmnt
/zmnt on /dev/vg01/lvol4 log,mincache=direct,convosync=direct on Fri Jun 2 12:55:00 2006


Rgds...Geoff

[TwoProc]

Re: Nelson question:

Nelson, you can change that number (multiblock_read_count)dynamically. So if they don't like what they see, you can simply put it back. Over here, this made a noticeable difference in total system throughput (not necessarily on single queries). It appeared that even though raising this factor created a higher bias of the cost optimizer to do full table scans, the fact that we had more efficient use of max sized data pushes on the buses to get the job done was actually a bigger benefit.

The only downside to the dynamic change is that effect of the change, while immediate for all future queries, isn't going to affect the costed plan of already running queries (which will remain more biased to full table scans, than index based scans). This means that the longest the effect could affect anything (pretty good rule of thumb huh? :-) ) is the length of the longest running query, and it would only affect THOSE long running queries. So, if after 30 minutes you've got only two jobs left over from before the change runnig, then those two jobs represent the total sum of still affected queries from the unwanted multiblock_read_count change.

As a way to manage the risk (and since it is dynamic), try changing half way the first time, and at a non-critical period. Try changing it to 12 first, and do it at an low-usage period. Using perfview (temp licenses can be easily obtained, and I DO recommend that you buy it if you don't have it -- for this very reason) you can do a comparison of total I/O and see if you did any good/damage. If there's not enough change to even tell, then you can progress to testing during a "medium activity" period (but put it back to 8 at high activity periods). Do that test for 2 or 3 days (even a week) and verify with PerfView, that at least you've done no harm. If things are gong well, then proceed with testing for high activity and then measure. If after the testing at 12 isn't hurting anything, then move to testing at a multiblock_read_count of 16, and do it incrementally, as suggested with the previous round of testing at "12".

P.S. Your application as an awdnkuwkoaos support affiliate has been accepted. :-)

[Alzhy]

John:

Thanks!

Where do I send the association dues?

;^)

[Geoff Wild]

Nelson,

I belive the dues are 1 pint - deliverable at the HP Tech Forum this year!

Rgds...Geoff

[TwoProc]

Geoff,

That speaks so near and dear to my heart (*sniff*, BLUBBER) that now FINALLY we now know what the association dues are...

We never knew what was fair and equitable before, and we are now so lucky to have a standardized association plan. We're so fortunate!

:-)

P.S. put me down for a Samuel Smith's Oatmeal Stout!

[Hein van den Heuvel]

It must have been too long a week for me or something but I fail to find two critical items in this topic:

1) where is the performance problem?
It seems like a nicely busy system, but nothing dramatically wrong.

2) what the heck does "awdnkuwkoaos" stand for?

For me, the most important and insightful replies in this series is the least 'rewarded' one:

>> John Joubert wrote...
"Geoff, IMHO - I wouldn't call the loss of 3 or 4 transactions (depending on how they are lost I guess, that is, error codes) at peak periods a performance issue. Sounds like a bug in Oracle, or possibly a resource constriant (in Oracle or in HPUX) to me. What's the error code that Oracle provides when the failure occurs."

Lost transactions are rarely a performance problem but really suggest a functional problem which may only get exposed under load.

Sure sometimes transactions get 'lost' due to poor performance. For example: a local store lost a transaction from me this morning when the checkout performance was so poor that I walked away.
But was this transaction really lost? Who is to know whether I really intended to buy that item? Where was it counted and then voided? :-)

I think it would be nice to know what defines a 'lost transaction' here. Two counters getting out of sync? Some error messages?
I would be really surprised to learn if the system 'could not keep up' and then ignored selected transactions. It is much more typical for work to pile up and queues to form. But I suppose there are real-time-ish systems where events come in and a system has only x-amount of time to react to it, or it will be gone.

If anyting I'd focus tuning on the logwriter and database writers.
Larger redo log and a larger logbuffer maybe appropriate, but your dba's reactions suggest they are pretty much on the ball there.
Anyway, those write activities may well cause a system to show overly busy while still being able to handle the load.
For example, the log writer starts to issues larger IOs, commiting more per IO when the response time from the IO subsystem gets less.
To put it simplistically... on a high performing IO system you may see 500 IO/sec @ 8KB = 4MB/sec and when the IO channel gets busy (for outside reasons) to can morph to 50 IO/sec @80KB... still 4MB/sec.

fwiw,
Hein
HvdH Performance Consulting.

btw... Geoff, when posting 'glance' data and the likes please consider marking the new "Retain format(spacing)." for best formatting. Then again... maybe you did, as after pasting into a notepad it looked nicely lined up.

[Steven E. Protter]

Shalom Geoff,

Very interesting discussion.

I'd consider decreasing the OS buffer even further than you have, with dbc_max and min close together. You don't want the OS making buffer cache changes during peak transaction times.

I think I saw that gzip is being used on the redo logs. If so, stop doing that, its expensive and causes problems. If the logs need to be zipped, ship them off to NFS and let some other system zip em.

I think your lun may be a little big, and you could benefit from a strategy that puts high i/o filesystems on smaller, more targeted LUNS. Thats probably not practical, but the configuration of the storage may be an issue that needs to be looked into.

SEP

[Geoff Wild]

Going from 8GB to 12GB of ram helped a lot - Oracle will be resizing the SGA.

I believe they will also be increasing the Oracle buffer cache.

The 100% disk I/O and CPU (becuase it is waiting on I/O) is what I'd like to see come down.

I don't know what the lost transactions are - this system updates price books and other things at gas stations.

The application servers are running on Windows...

Some LUNS are 32G and some are 8GB.

# vgdisplay -v vg20iprismp
--- Volume groups ---
VG Name /dev/vg20iprismp
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 11
Open LV 11
Max PV 24
Cur PV 24
Act PV 24
Max PE per PV 4315
VGDA 48
PE Size (Mbytes) 8
Total PE 55155
Alloc PE 54148
Free PE 1007
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg20iprismp/data01
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vg20iprismp/data02
LV Status available/syncd
LV Size (Mbytes) 12000
Current LE 1500
Allocated PE 1500
Used PV 4

LV Name /dev/vg20iprismp/data03
LV Status available/syncd
LV Size (Mbytes) 10000
Current LE 1250
Allocated PE 1250
Used PV 3

LV Name /dev/vg20iprismp/data04
LV Status available/syncd
LV Size (Mbytes) 187000
Current LE 23375
Allocated PE 23375
Used PV 8

LV Name /dev/vg20iprismp/indx01
LV Status available/syncd
LV Size (Mbytes) 191000
Current LE 23875
Allocated PE 23875
Used PV 19

LV Name /dev/vg20iprismp/exports
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 640
Allocated PE 640
Used PV 3

LV Name /dev/vg20iprismp/arch
LV Status available/syncd
LV Size (Mbytes) 24000
Current LE 3000
Allocated PE 3000
Used PV 6

LV Name /dev/vg20iprismp/redo01a
LV Status available/syncd
LV Size (Mbytes) 504
Current LE 63
Allocated PE 63
Used PV 1

LV Name /dev/vg20iprismp/redo01b
LV Status available/syncd
LV Size (Mbytes) 504
Current LE 63
Allocated PE 63
Used PV 1

LV Name /dev/vg20iprismp/redo02a
LV Status available/syncd
LV Size (Mbytes) 504
Current LE 63
Allocated PE 63
Used PV 1

LV Name /dev/vg20iprismp/redo02b
LV Status available/syncd
LV Size (Mbytes) 504
Current LE 63
Allocated PE 63
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t10d0
PV Name /dev/dsk/c6t10d0 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t10d1
PV Name /dev/dsk/c6t10d1 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t10d2
PV Name /dev/dsk/c6t10d2 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t12d2
PV Name /dev/dsk/c6t12d2 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t12d3
PV Name /dev/dsk/c6t12d3 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t12d4
PV Name /dev/dsk/c6t12d4 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t12d5
PV Name /dev/dsk/c6t12d5 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t6d1
PV Name /dev/dsk/c6t6d1 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t6d0
PV Name /dev/dsk/c6t6d0 Alternate Link
PV Status available
Total PE 4315
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d0
PV Name /dev/dsk/c10t3d1 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d1
PV Name /dev/dsk/c10t3d2 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d2
PV Name /dev/dsk/c10t3d3 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d3
PV Name /dev/dsk/c10t3d4 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d4
PV Name /dev/dsk/c10t3d5 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d5
PV Name /dev/dsk/c10t3d6 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d6
PV Name /dev/dsk/c10t3d7 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d7
PV Name /dev/dsk/c10t4d0 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d0
PV Name /dev/dsk/c10t4d1 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d1
PV Name /dev/dsk/c10t4d2 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d2
PV Name /dev/dsk/c10t4d3 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d3
PV Name /dev/dsk/c10t4d4 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d4
PV Name /dev/dsk/c10t4d5 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d5
PV Name /dev/dsk/c10t4d6 Alternate Link
PV Status available
Total PE 1088
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d6
PV Name /dev/dsk/c10t4d7 Alternate Link
PV Status available
Total PE 1088
Free PE 1007
Autoswitch On



And yes I did click on "retain format" :)



Stop the gzip eh? hmnnnn that would mean adding more disk space....

Rgds...Geoff

[Geoff Wild]

BTW -

# kctune |grep dbc
dbc_max_pct 7 7 Immed
dbc_min_pct 5 5 Immed


Rgds...Geoff

[TwoProc]

Hein,

"awdnkuwkoaos" got its first breath of overzealous sarcasm in the following thread:

https://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1007168

:-)

[TwoProc]

Geoff, just use RMAN to pick those up (archive logs) against your tape library solution, it's a lot faster and more convenient for recovery.