SAN Performance on RPXXX

[joe_91]

Team:

Our team feels that the SAN performance on our HP-UX servers is not optimal. What are the things that i need to check/monitor from the OS point of view? Please share..

Thanks

Joe

[Mridul Shrivastava]

Can try sar -d using the intervals but it will again add to the performance. If you have glance installed then this will be helpful in determining any performance issues.Example:

# sar -d 2 5

HP-UX shivalik B.11.23 U ia64 06/21/06

20:15:52 device %busy avque r+w/s blks/s avwait avserv
20:15:54 c0t2d0 0.50 0.50 1 18 0.00 10.10
20:15:56 c0t2d0 2.00 0.50 2 40 0.00 14.15
20:15:58 c0t2d0 3.00 0.50 6 104 0.00 8.42
20:16:00 c0t2d0 0.50 0.50 0 8 0.00 10.27
20:16:02 c0t2d0 0.50 0.50 0 8 0.00 9.42

Average c0t2d0 1.30 0.50 2 36 0.00 10.01

For more details refer map pages of sar.
But this is very basic thing to start with.

[Steven E. Protter]

Shalom Joe,

http://www.hpux.ws/system.perf.sh

SEP

[Alzhy]

Stop 1:

Check your array(s). If it has historical data - then observe if your HP-UX servers have actually swamped it at any time in the past.


Stop 2:

If the answer to stop 1 is a big NO (i.e. you have an XP storage array and Perfromance Advisor shows your ACPs, and your CHips have very low usage historicals/currents), check your HP-UX usage of the storage objects from the SAN storage(use SAR, iostat, vxfsstat, glance, etc.). Check for queuing (sar -d avqueue primarily). If it all shows as healthy, you can conclude either of the following:

a. Your applications (Database, etc) is not really optimized to get the most out of your storage. Review and re-layout (i.e. stripe your storage volumes)

a.1 Your application is not optimized to get th emost storage juice out of your storage. Check it's tunable parameters relating to I.O.

b. COnsider using RAW storage (if your server load is a Database)

c. Tweak the filesystem (if cooked Oracle storage - i.e. enable DirectIO and minimal cacheing..)

d. Check your kernel parameters relating to IO -- i.e. scsi_max_qdepth, scsi_max_phys, vol_maxio, vol_max_parallelio, etc..


Stop 3.
Check your SAN switch ports for oversuscription, ISL paths, FC path bandwidth issues (very rare).


Hope this helps..

[joe_91]

out of the three machines on one dmesg|grep -i phy does not return anything (just the prompt)..

pl find kmtune out put from one machine

# kmtune|grep -i io
Parameter Current Dyn Planned Module Version
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
eisa_io_estimate - - 0x300
io_ports_hash_locks 64 - 64
iomemsize - - 40000
netisr_priority - - -1
region_hash_locks 128 - 128
vol_default_iodelay 50 - 50
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
# kmtune|grep -i scsi
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 -

1048576
Thanks

Joe

[Victor BERRIDGE]

Hi Joe,

Not knowing how you connect to what type of SAN makes it difficult to give an answer, for instance:
what/how many HBAs are you using - in which mode?
Did you tune the PV/VG accordingly (queue depth /IO Timeout ) to your Bays specs?
Are you using a load balancer?
If pure LVM ave you tried striping and alternate the disks?
What is your throughput?

All the best
Victor

[Victor BERRIDGE]

Hi again Joe,
I just stressed my poor RP5450:
look at the attached file, it will give you some ideas of IO performance on RPXXXX

[Victor BERRIDGE]

And this is what I did:
octant:/sas9wks $ du -sk sas9cds/
8304632 sas9cds/
octant:/sas9wks $ timex cp -rp sas9cds/ sas9cds_copy

real 3:03.86
user 0.87
sys 2:18.80

octant:/sas9wks $



All the best
Victor