IO Perfs on Montecito server (RX6600)

[Jean Mesquida_1]

Hi all !

I am running Iozone on my newly installed servers and get very low performances when record size is 256K or more. This is the case when accessing either internal SAS disks or external EMC array.

My conf is HP-UX 11.23 MCOE 09/06 with up2date patches.

Iozone has been compiled on these servers.

Did some of you experience the same behaviour ?
Thanks for your help
Jean

[Alzhy]

Interesting.

Can you describe how you're testing? The nature of the filesystems and their underlying volumes?

[rick jones]

ISTR that VxFS will use "direct I/O" when the I/O size is 256KB or larger. This means that the DMA is performed directly to/from your application buffers, which means the application becomes blocked during the I/O and opportunities for read-ahead and/or write-behind become, well, very limited.

Alas, I cannot recall what the magic command is to change that value - something involving vxfstune or the like?

[Alzhy]

Rick,

Are you surmising "Discovered DirectIO" ? There is indeed a tunable for that... I will post once I recall definitely what it is.

However, iozone features tests for RAW device IO.. Our friend should test that as well.

[Jean Mesquida_1]

Thanks for your valuable answers ! I didn't know the information you stated, Rick.

Please notify when you remember the right tunable. I'll also try to find this information.

Thanks a lot.
Jean

[Jean Mesquida_1]

BTW, I am not nearby these servers right now and cannot perform the raw devices tests provided by Iozone.
My Oracle databases will be installed on VxFS filesystems.

[rick jones · Accepted solution]

$ man -k tune
gettune(2) - get the value of a kernel tunable parameter
kmtune(1M) - query, set, or reset system parameters
settune(2) - set the value of a kernel tunable parameter
tunefs(1M) - tune up an existing HFS file system
tunefstab (vxfs)(4) - VxFS tuning parameters table
tuneinfo(2) - retrieve detailed information about kernel tunable parameters
vxtunefs(1M) - tune a VxFS File System



vxtunefs(1M) vxtunefs(1M)

NAME
vxtunefs - tune a VxFS File System

SYNOPSIS
/sbin/vxtunefs [-ps] [-f tunefstab] [-o parameter=value]
[{mount_point|block_special}]...

...

The VxFS tuneable parameters are:

...

discovered_direct_iosz
Any file I/O requests larger than the discovered_direct_iosz are
handled as discovered direct I/O. A discovered direct I/O is
unbuffered like direct I/O, but it does not require a synchronous
commit of the inode when the file is extended or blocks are
allocated. For larger I/O requests, the CPU time for copying the
data into the buffer cache and the cost of using memory to buffer
the I/O becomes more expensive than the cost of doing the disk
I/O. For these I/O requests, using discovered direct I/O is more

Hewlett-Packard Company - 2 - HP-UX Release 11i: November 2000

vxtunefs(1M) vxtunefs(1M)

efficient than regular I/O. The default value of this parameter
is 256K.


I have also heard, but do not know where to go to confirm, that things like MSA arrays will bypass their own caches for "really large I/O" - also > 256 KB. It might be a good idea to check the specs of the storage you are using to see if it does that too.

[Jean Mesquida_1]

You got it !

I changed this parameter (discovered_direct_iosz) on my test servers and the Iozone results dramatically improved.

I'll try to know what value was applied on filesystems before the September 2006 release of HP-UX 11.23.

Thanks a lot for your help.
Jean

[Jean Mesquida_1]

With March 2006 release of HP-UX 11.23, I had these parameters for VxFS filesystems :

Filesystem i/o parameters for /test

read_pref_io = 65536
read_nstream = 1
read_unit_io = 65536
write_pref_io = 65536
write_nstream = 1
write_unit_io = 65536
pref_strength = 10
buf_breakup_size = 131072
discovered_direct_iosz = 262144
max_direct_iosz = 1048576
default_indir_size = 8192
qio_cache_enable = 0
write_throttle = 0
max_diskq = 1048576
initial_extent_size = 8
max_seqio_extent_size = 2048
max_buf_data_size = 8192
hsm_write_prealloc = 0
read_ahead = 1

Today, with 09/2006 release of HP-UX 11.23, I have the same (values for these parameters are identical in 03/2006 and 09/2006), plus these new ones :

inode_aging_size = 0
inode_aging_count = 0
fcl_maxalloc = 65075200
fcl_keeptime = 0
fcl_winterval = 3600
oltp_load = 0

The I/O behaviour for filesystems created with these 2 versions of HP-UX seems to be very different.

[Alzhy]

Jean,

There should as well be a document on tunuing OJFS (aka VxFS) somewhere on HP's websites. In that document, these parameters are fully described so you can tune accordingly.

Have you tried at all comparing RAW performance versus your adjusted filesystems (for discovered DirectIO)? It would be very interesting indeed to find out the differences.

Thanks.

[Jean Mesquida_1]

Nelson

I have not been able to perform Raw IO tests yet (maybe at the end of this week).

I could not find any information concerning the new FS parameters. If you did, please provide the corresponding URL.

Thanks
Jean

[Juan_20]

Hi

Do you solve this problem Jean ??
Same problem here.

[Jean Mesquida_1]

Hola Juan !

This problem seems to be met whenever a new version of Veritas VxFS is installed (on HP-UX or Sun Solaris). The "solution" is to increase the discovered_direct_iosz to, say, 10 Mo : then the behaviour looks more like it was one year ago... Use vxtunefs to do this.

HTH
Jean