Suggestions on moving folders from internal disk to SAN(11.31,rx7640)

[Deepak Seth_1]

we are experiencing some performance issues and after analyzing the data , the biggest culprit is the high i/o waits on boot disks. Nothing much can be done to quickly improve the waits . Here is my plan
- move /var/adm/syslog to SAN - ??? , suggest ?
- move /var/opt/perf to SAN - i think this is definately a good idea .

Any other suggestions. Better way to dealing with this kind of issue (with out adding more h/w )

hardware specs
rx7640 - 3 vpars (problem is in 1 vpar only) , 11.31 , Sept ,2009 patch bundle , 2 HBA's , 146 GB drives 15 K RPM .

[vishnu.khandare]

Hi Deepak,

As you said the problem is high i/o,
Which process are on top, check its priorites
U need to check what the big files then try to perf that files. tak backup/ reomve.
Its gud to hav this system dir on VG00 root disk. Dont move it on SAN.

Its better to move data mount points on SAN from root VG.

Regards
Vishnu Khandare

[Tim Nelson]

I really doubt that the io to those two files is crippling your performance.

how did you determine that io to /var/adm/syslog/syslog and /var/opt/perf/datafiles is hindering your performance of your application.

and what is your application and where is its data located ? same disks ?

[Deepak Seth_1]

Vishnu - its oracle and ERP processes which are SAN file systems .
Tim - All oracle/ERP located on SAN (clariion ) . I am using the Native multipathing (ALUA) as well.

don't know whether /var/adm/syslog or perf is hindering but just 2 things which
are mostly active in general.
Also , don't know whether this exercise will improve anything . HP recommendation is add more paths (which needs money/downtime and so many approvals)

attaching the sar -d output using 5 sec data for avque > .50 and await > 5 .

[Tim Nelson]

yes those stats are terrible.

without knowing what the config/topology is I can only assume.

if you wish, provide some more info.

what is disk41, disk55 and disk59 ?

is disk59 some huge lun off of the clarion ?

how busy is the clarion ?

[Deepak Seth_1]

disk41 & disk55 are boot disks

disk59 is SAN carved out of RAID Grp with 7 300 GB FC 15 K drives . I can definately see its being very busy but majority of time the avg service time and avg waits are fine .I did some load balancing few weeks back by moving some of the heavely used oracle data files to another LUN (from different RAID grp) - disk75.

But more concern about how to fix this heavy i/o load on boot drives ?

This system have 32 Gb of memory and 8 CPU , so no worries on basic resouces.

lvmtab

/dev/vg00
/dev/disk/disk41_p2
/dev/disk/disk55_p2
/dev/vg01
/dev/disk/disk59
/dev/disk/disk75
/dev/vgora
/dev/disk/disk66
/dev/vg02
/dev/disk/disk80
/dev/vgswap
/dev/disk/disk86

[Deepak Seth_1]

clariion is about 60-70% busy. With i/o / sec as high as 500-800 for this particular LUN. The avg i/o load is around 10K i/o's per second.

[Bill Hassell]

> don't know whether /var/adm/syslog or perf is hindering but just 2 things which
are mostly active in general.

syslog? Files in that directory are almost idle compared to other activities on the system. Your syslog log files (all of them) would have to be growing at multi-gigabytes per day to have any effect on overall disk I/O. /var/opt/perf can be a bit busy, but again, you'll not see dozens of writes going to the logfiles in the perf directory.

Since you have Glance/mwa, look at the I/O by filesystem (type the letter i while glance is running). Phys I/O rates are the most important and need to be tracked down to the actual file(s). Most likely there are some temp files in /var causing a lot of I/O.

[ManojK_1]

Do you have glance in your system?
If it is there try with "glance -v" and check which lv is having high I/O.

Application binary is there in rootvg?

Does the local Disks are mirrored?

Manoj K

[Rudolph Wilson]

Several of our Commercial Off The Shelf applications (Ab Initio and SAS for example) have temporary files written to /var by default. Moving their directories to SAN can definitely speed up application performance. Check the documentation of all of your applications.

[Deepak Seth_1]

Bill / Manoj/Rudolph - Thanks for the reply.

I did look many time i/o by FS , but how to find the top processes for high phy i/o ? All
the time the top system processes are for oracle / erp application which is on SAN anyway .

Also is there way to collect glance data since this server is used by Asia and they usually comes
to work late evening.
Boot disk is hardware mirrored - so no overhead from o/s side

application logs/tmp files are going to SAN file system - confirmed

If i could figure out a way to find the top high i/o processes for all boot file systems , then
at least i can troubleshoot it better.

Any way to do that .

We do have BMC Patrol also , but can't be of much use either.

[Rudolph Wilson]

Try '/opt/perf/bin/extract -gapdznuy -xp d-1 -r /vat/opt/perf/reptfile' command to create performance metrics text files for the previous day (creates in your current directory). The xfrdPROCESS.asc file shows processes that exceed limits set in config file. Look for 'D' (disk) in the "Interest" column. It won't tell you what filesystem had high i/o, but it is a start. Then you can look at xfrdGLOBAL.asc and xfrdDISK.asc for those specific time frames to compare.

Could be tedious, but it is a start.

To look at other times, the '-s mm/dd/yy hh:mm' and '-e mm/dd/yy hh:mm' switches can be used to look at data from other timeframes.