Need help troubleshooting performance issue

[Raj D.]

Correction:
bumped 30% to 70% --> to be read as "30% to 100% "
[That means 70% sudden increase of cpu reources.]

[Dennis Handly]

memory 524023 124095 399928 24%
total 1251063 691602 559321 55%

You actually have 524 Gb of memory and an extra 700 Gb of device swap?

You probably should remove lots of that device swap.

[Michael Steele_2]

Note: you attached your collection script but not the sar -d date - only provide the TOTALS please!!!!!

So you've got a process bottleneck, a cpu bottleneck and a disk bottleneck. But which process caused it?

[Michael Steele_2]

16:35 - disk bottleneck
16:50 - cpu
17:05 - free memory drops from 651 x 10**6 MB
-to- 9 x 10**6 MB -or- about 85% less of normal if 651 is normal.

So the first thing to happen was a disk bottleneck, based upon you MWA data.

Since paging jumped astronomically this explains the disk bottleneck.

A high priority page in request will cause the processing to stop until the page is found.

This is most certainly an application issue -

What in the application causes high priority requests?

What was running / happening / were the users doing at 16:45 ?????

Note: 16:45 - end of day - some monster report / select statement

[Michael Steele_2]

Hi

I am thinking this is a global select statement. Why? Its not just one high priority request, its a lot of high priority requests. So many that memory filled up and still incomplete, still lookiing for more, when the box crashed.

I am also thinking this was run from a power user in SAP. It fits. Basic users aren't going to have the high priority privileges. But you can verify this with SAP as to user priorities.

And that leaves and SAP admin. And its going to be a fight to get it out of a college.

[Tony Williams]

Thanks everyone for all of the help. We were able to pin down the problem to an Online database backup that only runs on Fridays and wasz moved a few hours up. The part of the job that put the Oracle tables into backup mode coincides with other normal processing that occurs and the result was too much disk consumption, too much CPU, etc. We moved the backup job.