rp8400 - 11.11 - root disks show 100% used

[moonchild]

need help in finding why root disks are showing 100% used in both sar -d and glance.

we're experiencing performance degradation in running commands that are hanging. We do not have any errors in syslog that point to a disk failure.

[moonchild]

here's a syslog attached

thank you in advance

[Jeeshan]

Hi

the 2 root disks(c0t6d0 and c2t6d0) is in critical condition. In my previous experience i have seen such type of usage cause my OS crash. So make immediate decission to case a lock in HP support. In this position if you reboot your server, it may not up the OS.

From your side you can ensure you have latest ignigte recovary tape.

[Khairy]

hi,

from the sar output, it seems that your disks is 100%busy, not 100% used. It occured to 2 internal disk c0t6d0 and c2t6d0. Both are 73.4Gb disks.

more info are needed to identify the problem.

I'm suspecting some processes r busy writing on the disks and that may cause the wait IO more that 60% continuously.

But if i may guess, are c0td0 and c2t6d0 are mirrored via LVM?
You can check this by posting output of `vgdisplay -v vg00`. At the end of the output you shall see the disks. Check for number of PE with LE. In mirrored vg00, number of PE is alway double that LE. Eg: LE=100, PE=200.

I guessing you're running oracle. Do you put your archives,redo,undo logs in vg00? If you do, that may be starting point for further investigation.

Let us know more about your system setup.

Rgds

[moonchild]

Yes those two internal disks are mirrored.


booting the system comes back fine.

Disk Utilization was varying between 80-95% before making the DBs up. After reboot we started the DBs, it has reached 100% and it was not at all varying.

I am attaching the fstab where you cna see the oracle filesystem on vg00. we have oracle, logs and auidits.

Thank you for your help

[TTr]

This is way to much I/O to put on a single disk (mirrored pair). You need two get additional storage, at least two additional disks (mirrored pairs) and preferably additional I/O.

You did not mention what kind of server it is but if you have room to add internal disks you may still have the I/O problem because with adding internal disks the I/O traffic stays in the same I/O controller.

It sounds that you are on a tight budget having to use the internal OS disks for a database but at a minimum you shopuld add another I/O interface with external storage.

[Court Campbell]

I think you already answered your problem to your IO issue. You definitely need some sort of external storage to put your oracle data files on.

[Eric SAUBIGNAC · Accepted solution]

Bonsoir Fils de la Lune,

As many said : too many FS for a single disk. You should consider moving most of them to the EMC array : good stats from sar -d.

Maybe you have also insufficient RAM issues that could add some more stress to your internal disks ? What is output from "vmstat -S 5 120" (it will take 10 minutes to achieve) ? Look at columns "po" and so : if po is often a number with 2 digit you have memory bottleneck. If "so" is often > 0 it is not a memory bottleneck but a memory SOS (don't know how to say in english ;-)

An other point that has no direct link with your question but that you should consider. Maybe it is an issue you have control on, maybe not :

- in syslog I found : "ct_query failed. hw_path = 0/0/2/0/0"
- in your ioscan -C disk -fn[ku] we can't see any disk under 0/0/2/0/0. Why ?

--> I guess this fiber channel card is missconfigured on the SAN switch : zoning, disabled port, etc ... ?

Regards

Eric

[moonchild]

vmstat output is fine no po.

Thanks all

[Khairy]

hi moonchild,

vmstat unveils no memory performance prob. This is truly IO issue.

Migrate all oracle related filesystem from vg00 out to external disks (if you have). Go for raid 1+0 for filesystems contains your archive/redo/undo logs.

I'm guessing this contain the oracle archives/redo/undo logs.But for confirmation, you could check with your DBA.
/dev/vg00/fs_oracle_log /var/opt/oracle/log vxfs delaylog 0 2

and maybe some other filesystem that your apps maybe accessing frequently for intense IO operation like:
/dev/vg00/fs_adm /usr/local vxfs delaylog 0 2
/dev/vg00/fs_emc /opt/emc vxfs delaylog 0 2
/dev/vg00/fs_opt_ctsa /opt/ctsa vxfs delaylog 0 2
/dev/vg00/fs_crash /var/adm/crash vxfs delaylog 0 2
/dev/vg00/fs_perf /var/opt/perf vxfs delaylog 0 2
/dev/vg00/fs_sw /var/adm/sw vxfs delaylog,nosuid 0 2
/dev/vg00/fs_mar_mig /mar_mig vxfs delaylog,nosuid 0 2
/dev/vg00/fs_prod /prod vxfs delaylog 0 2
/dev/vg00/fs_openspool /opt/openspool vxfs delaylog 0 2
/dev/vg00/fs_controlm /opt/controlm vxfs delaylog 0 2
/dev/vg00/fs_ctmagent /opt/ctmagent vxfs delaylog 0 2
/dev/vg00/fs_patrol /opt/patrol vxfs delaylog 0 2
/dev/vg00/fs_ftp /home/ftp vxfs delaylog,nosuid 0 2
/dev/vg00/fs_irm /var/opt/oracle/IRM vxfs delaylog 0 2
/dev/vg00/fs_openv /opt/openv vxfs delaylog 0 2
/dev/vg00/fs_opt_esm /opt/esm vxfs delaylog 0 2

If you have another external disks for use, create a new volumegroup and migrate the filesytems above.