sar -d output showing 100% busy disk and rx5670 process very slow

[Khairul_1]

Hi,
we have HP RX5670 Cluster Server 8GB RAM with VA7110+DS2405 External storage running oracle 9i Database for online Telecom Billing production.

We are getting DBA complien that system and IO very slow. we have check and found that sar -d output showing some disk 100% busy.
I am attaching the output. Please help us how we can solve it? Do we need to increase the memory?
Thanks
Khairul/Bangladesh

[Jeeshan]

>>we have HP RX5670 Cluster Serve

is this is only one node?

check your dbc_max_pct and dbc_min_pct kernel parameter settings.

you may set it to min-2 and max-10.

[Asif Sharif]

Salam Khairul,

Please post the output of the following commands.

kctune |grep dbc_max_pct
kctune |grep dbc_min_pct
strings /etc/lvmtab

Regards,
Asif Sharif

[Sandeep_Chaudhary]

which VG is using following disks. Is it used for swap or oracle


Average c7t0d1 82.00 15.95 831 13303 5.89 5.34

Average c7t0d4 91.44 20.40 778 12448 29.67 5.23

Average c7t0d6 49.78 9.39 26 411 6.70 39.23
Average c7t0d7 40.44 21.41 621 9940 6.64 2.61

Average c7t1d1 29.00 15.70 126 2009 9.10 10.92
Average c7t1d2 65.89 7.39 131 2091 4.81 15.88

Average c7t1d4 103.00 11.67 637 10192 4.99 4.40
Average c7t1d5 31.56 17.28 209 3351 32.97 11.49
Average c7t1d6 36.56 5.31 124 1977 3.57 6.87

Average c7t2d0 34.11 12.90 142 2277 7.44 6.59

[Jeeshan]

also post the output of

#sar -b 5 5

#sar -v 5 5


in VA check the IOPs of disk drives.

[Khairul_1]

Thanks to all of you.
As per Oracle Metalink suggessation we need to increase following kernel parameter about 6months ago?
dbc_max_pct 25 25 Immed (previously 10)
dbc_min_pct 5 5 Immed (previously 2)

The all LUN are one vg and Production Database on External SAN Storage VA7110.

I am attaching the vgdisplay -v /dev/vgunicorn, sar -d , sar -b, who, ps -ef, and kctune output for your information.

Please help us how we can solve the issue of slow IO and busy 100% disk?
Do we need to increse RAM from 8GB to 12 or 16GB?
Thanks
Khairul/Bangladesh

[Jeeshan]

look closely to your sar -b output.the read cache and write cache is abnormally low, which means your system is not using the buffer cache properly.

tune the dbc_max_pct to 10 and check the impact and usage using
#sar -b 5 5

[SKR_1]

What is the memory utilization right now.
If its also 100% utilized at this time on daily basis, then you can think upon to extend the memory. Also please check the timeout setting of the disks which are highly utilised.

you can set the timeout to 60.

Thanks

SKR

[Duncan Edmonstone]

Well one point I see here is that all your PVlinks atre using the same controller on the VA (c7) so there;s no IO going to c9.

II never used VA disk arrays much, but I'm pretty sure you could use both controllers at the same time, so you should look to see what LUNs are owned by what controller and then make sure that your have half owned by each. You then need to reflect that configuration in your LVM configuration (i.e. the primary PVLink goes to the owning controller).

HTH

Duncan

[Duncan Edmonstone]

>> Well one point I see here is that all your PVlinks atre using the same controller on the VA (c7) so there;s no IO going to c9.


I did of course mean c5, not c9

HTH

Duncan

[Khairul_1]

Hi,

Is it required any reboot the system or backup OS and Database for change kernel parameter dbc_max_pct?

Thanks
Khairul

[Jeeshan]

no, as your OS is 11iv2

[Hein van den Heuvel]

Thanks for sharing the data (notably) sar in .TXT form. Good stuff, allthough the window is a little short as proven by the two average being vastly diffent. You may have hit a checpoint window in one of them.

Forget about dbc_min and dbc_max.
That's just there to keep the clueless folks occupied.
It serves no serious performance function for an Oracle application.

Having said that, your system seems to be using all memory (a good thing!) but be careful that the dbc_??? tinkering crowd did not make you raise dbc_min too high as you may force the system to reserve memory for no good reason (as proven by the sar -b output).

>> We are getting DBA complien that system and IO very slow.

Engage the DBA.

Let him/her explain what this is based on.

They have the knowledge!

Oracle keeps wonderful stats, have the DBA interpret those for your benefit.


The 13:08 data suggest a pretty intense IO load. The sum is more than 6000 IO/sec and 100,000 blks/sec. That's serious. At that oint you need to know exactly how many spindle, what connections and so on are being used.
What was the system designed to handle?
Is the current IO load 'reasonable' from that design perspective?

The storage system is holding up prety good (avserv) but heavily taxed (avque).
It may well be that the system needs more memory for the oracle SGA.
Seems to me the SGA as about 4GB currently, out of 8GB physical no?
Looks like it could use soem more, but only the Oracle stats can confirm this.

The Oracle stats, in the hands of a capable DBA, can also tell (the DBA) whether Oracle (query) tuning may be helpful.

Without anything to proof this, just a hunch, I would start working with the DBA to make sure the checkpoint settings are reasonable. Checkpointing too often can generate a tremendous, and avoidable IO load on all devices involved.

Don't accuse... stimulate thinking.
Don't defend... question and help explain. For example: "my system tools observe periods of 2000+ r+w/second on c7t0d6 (which maps to /xxx) is that reasonable for the Oracle load demanded at that time?"

Hope this helps some.
Hein van den Heuvel
HvdH Performance Consulting

[Khairul_1]

Hi Thanks for your suggessation.
We are getting attached information from Metalink.

[Torsten.]

Since the VA7110 arrays are a bit old now, think about the possibility of a failed cache battery. If such battery has failed, this may slow down the array dramatically.

Please post

# armdsp -a

[Khairul_1]

Hi Torsten,

amrdsp -a san-va output is ok.
I am attaching the output.

Could you or any one please help us how we can solve slow IO issue as per Oracle DBA & Metalink?

Do we need to increase the RAM for increase the DB_Block_Buffers?