Oracle backup consumig excessive CPU time

[Michael Steele_2]

Raid 5 devices aren't mirrored and CU indicates "...RAID..." device. (* JBOD's are mirrored. *)

Anyway. Time to move on. Can't make presumptions.

(* Why don't you come to NYC for your get together Paula? Airfares just as cheap heading west over the Atlantic. Great gambling over here at Foxwoods and Mohegan Sun. *)

[Paula J Frazer-Campbell]

Michael

CU did not say which raid level and as you know raid 0+1,1 and 10 are mirrored.

;^)


NYC would be great, I do not gamble as I know how easy it is to cheat -

Why not try and organize something - perhaps just before thanks giving?

I would be interested in attending.


Paula

[Randy Navarro_1]

We're running RAID-1/0. Also attached is a partial sar output.

You'll see the backup starts at 2300. At 35 seconds after CPU load begins increasing. And, at 23:01:40 CPU ocassionally reaches 0% idle time.

[Michael Steele_2]

Thanks. I???ve reviewed your sar report and I'm concerned with your %wio numbers. They seem to indicate a tape device bottleneck.

Can you also attach 'swapinfo -tam' and 'vmstat 5 5'. You may also be paging out. Note the value of 'po' in vmstat. Is it non zero?

%wio is the percentage of time that processes have spent waiting for I/O and anything above 15 is bad.

You've got numbers repeatedly approaching 100% wait time.

You have several instances of jobs waiting 80% of the time or higher, with 96% the highest at time index 23:08:10.

Either your pipe into your device is too small or you need bigger tape devices.

Regarding the LVM split. In my opinion you will see no performance advantage since the problem is at the other end of you I/O and not at the beginning where an LVM split would provide performance enhancements.

What is your I/O connection into your tape drives? Can you double them?

[Paula J Frazer-Campbell]

Hi

From:-

23:01:40 1 2 97 0
23:01:45 1 2 97 0
23:01:50 1 1 92 5

Notice that the usr and sys are very low and wio is high, this is a bottleneck from either the disks or to the backup drive.


Paula

[Randy Navarro_1]

Mike,

Attached is the system data from last night's backup that includes the vmstat and swapinfo. The file was huge so we pared it down to the first 7mins or so. That is representative of the CPU loading problem.

The backup was scheduled for 2300 last night and we captured data starting at 22:59, so you can see the "before" and "after" situation.

Also, file viewability is best in Wordpad.

Thanks,

Randy

[Bill Douglass]

What type of RAID setup do you have? Based on the wio times, and the fact that you're doing a simple cp of the dbf files, I wonder if

1) You're using s/w RAID, and it is causing a lot of waiting

2) You have a problem on your I/O bus.

Do you have any error messages in syslog.log for the period of the back-ups?

[Michael Steele_2]

You're not paging and you have enough swap, this is all about your tape device and the I/O pipe into it.

There is a possibility of a disk bottleneck, please attach:

sar -d 5 5

...during backup.

(* Just a few samplings please. *)

Is this a network or SAN I/O pipe into your tape devices? Please indicate with speeds. For example, 100BaseT, 1 gb / second, etc.

Also, please check LOGTOOL for I/O HW failures. This is a historical log and doesn't have to used during the backup.

STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > VIEW > RAW SUMMARY.

Note the first and last dates of transactions and calculate the difference. If the difference is short, like 4 hours, then this is important to note. Now read down the report of hardware addresses and observe the integer numbers in parenthesis. Anything over 150 in this 4 hour period should be called into HP for replacement.