Suggestions on moving folders from internal disk to SAN(11.31,rx7640)

[Deepak Seth_1]

we are experiencing some performance issues and after analyzing the data , the biggest culprit is the high i/o waits on boot disks. Nothing much can be done to quickly improve the waits . Here is my plan
- move /var/adm/syslog to SAN - ??? , suggest ?
- move /var/opt/perf to SAN - i think this is definately a good idea .

Any other suggestions. Better way to dealing with this kind of issue (with out adding more h/w )

hardware specs
rx7640 - 3 vpars (problem is in 1 vpar only) , 11.31 , Sept ,2009 patch bundle , 2 HBA's , 146 GB drives 15 K RPM .

[vishnu.khandare]

Hi Deepak,

As you said the problem is high i/o,
Which process are on top, check its priorites
U need to check what the big files then try to perf that files. tak backup/ reomve.
Its gud to hav this system dir on VG00 root disk. Dont move it on SAN.

Its better to move data mount points on SAN from root VG.

Regards
Vishnu Khandare

[Tim Nelson]

I really doubt that the io to those two files is crippling your performance.

how did you determine that io to /var/adm/syslog/syslog and /var/opt/perf/datafiles is hindering your performance of your application.

and what is your application and where is its data located ? same disks ?

[Deepak Seth_1]

Vishnu - its oracle and ERP processes which are SAN file systems .
Tim - All oracle/ERP located on SAN (clariion ) . I am using the Native multipathing (ALUA) as well.

don't know whether /var/adm/syslog or perf is hindering but just 2 things which
are mostly active in general.
Also , don't know whether this exercise will improve anything . HP recommendation is add more paths (which needs money/downtime and so many approvals)

attaching the sar -d output using 5 sec data for avque > .50 and await > 5 .

[Tim Nelson]

yes those stats are terrible.

without knowing what the config/topology is I can only assume.

if you wish, provide some more info.

what is disk41, disk55 and disk59 ?

is disk59 some huge lun off of the clarion ?

how busy is the clarion ?

[Deepak Seth_1]

disk41 & disk55 are boot disks

disk59 is SAN carved out of RAID Grp with 7 300 GB FC 15 K drives . I can definately see its being very busy but majority of time the avg service time and avg waits are fine .I did some load balancing few weeks back by moving some of the heavely used oracle data files to another LUN (from different RAID grp) - disk75.

But more concern about how to fix this heavy i/o load on boot drives ?

This system have 32 Gb of memory and 8 CPU , so no worries on basic resouces.

lvmtab

/dev/vg00
/dev/disk/disk41_p2
/dev/disk/disk55_p2
/dev/vg01
/dev/disk/disk59
/dev/disk/disk75
/dev/vgora
/dev/disk/disk66
/dev/vg02
/dev/disk/disk80
/dev/vgswap
/dev/disk/disk86

[Deepak Seth_1]

clariion is about 60-70% busy. With i/o / sec as high as 500-800 for this particular LUN. The avg i/o load is around 10K i/o's per second.

[Bill Hassell]

> don't know whether /var/adm/syslog or perf is hindering but just 2 things which
are mostly active in general.

syslog? Files in that directory are almost idle compared to other activities on the system. Your syslog log files (all of them) would have to be growing at multi-gigabytes per day to have any effect on overall disk I/O. /var/opt/perf can be a bit busy, but again, you'll not see dozens of writes going to the logfiles in the perf directory.

Since you have Glance/mwa, look at the I/O by filesystem (type the letter i while glance is running). Phys I/O rates are the most important and need to be tracked down to the actual file(s). Most likely there are some temp files in /var causing a lot of I/O.

[ManojK_1]

Do you have glance in your system?
If it is there try with "glance -v" and check which lv is having high I/O.

Application binary is there in rootvg?

Does the local Disks are mirrored?

Manoj K

[Rudolph Wilson]

Several of our Commercial Off The Shelf applications (Ab Initio and SAS for example) have temporary files written to /var by default. Moving their directories to SAN can definitely speed up application performance. Check the documentation of all of your applications.