very poor performance, prealloc command

[James R. Ferguson]

Hi:

> Steven: Maybe an OS patch will help with malloc.

This is isn't 'malloc' [which allocates memory] but _prealloc_ which allocates DISK.

...JRF...

[Laurent Menase]

Indeed Hein you are right, they ask a perf consultation to HP support or services, but not for support.

[Dennis Handly]

>Laurent: prealloc is better than dd on the fact it writes 8kb blocks from kernel, without needing to copy data to user level. It is a syscall.

Where do you get this strange idea? prealloc(2) isn't a syscall. As tusc shows, it calls write(2) and fsync(2).

[Laurent Menase]

in fact my mistake comes from the fact it is defined as a syscall prealloc(2) --> 2 means syscalls
and prealloc vnode operation exist.



So I may have made a bad asumption that prealloc(1) was using prealloc(2) syscall.

but indeed libc prealloc is using write/fsync/fseek

[Dennis Handly]

>Laurent: 2 means syscalls

Yes, intro(2) seems to wave its hands. But prealloc(2) is a libc function.

[Rajeev jain]

Thank you all for the input. To move away from the difference between the command used between Sun and HP (prealloc and mkfile), I used cp command this time.

What I wanted to see were the results from your environment.

There is a case sitting with Support and Performance Engineering.

After running a simple cp command within the same file system, HP returns in 2min 6 seconds and Sun in 15 seconds.

There is no app OR database on the servers, these are brand new servers connected to a brand new AMS200 for testing. Both Sun and HP servers have 32GB memory and 4 CPUs.
The disks/disk group/volume and filesystem is configured exactly the same.

The command used is $timex cp test test1
Where test=1G file.

[Rajeev jain]

I ran prealloc and mkfile on local root mirrored disks. HP and Sun both takes between 10-12 seconds.

cp takes about 14 seconds on each system.

[Hein van den Heuvel]

Maybe the test on sun had the input still cached, and hp not yet cached? The test file is _likely_ to have been cached as part of its creation, but you never know...

Anyway, your new test now measures the time it takes to read a file, as well as write (a copy), as well as measuring the ability of the system to cache that input file and retain it. The test has no control over where the cache stops and the IO starts and without any indication how much time is spend in the reading versus the writting.

Seems to me you took a step further away from reality and relevance. Unless of course the purpose of the system is to copy 1 gb files around. In that case you have constructed a perfect test, and would need to move on to the next phase: understanding what is wrong and fix it.

Did you check the file system cache min/max?
If you re-read my earlier reply it proved to potentially cause a 10x speed difference.

Due to past experiences, which are no longer valid and never were very valid, a good few HP-UX administrator like to severely cripple their systems by setting those too low. They'd pick values in the 2% - 5% of memory range, notably for systems which are expected to run Oracle.

Admittedly, that should still be enough to suck in a 1 GB file when 32 GB memory is present. But NOT when you also have to read in the input file. Check it (kctune)

Suggestions.

* Measure the reading part ... and load in memory.

- umount /test # flush cache
- mount /test
- time cp /test/file /dev/null # initial read
- time cp /test/file /dev/null # from cache?!

* Measure the writing part

- prealloc ?
- from /dev/zero with dd commands outlined earlier.
- from a MEMORY file system

* Now measure the cp. twice.

( Dennis, did truss show alternating writes and fsyncs, or one fsync at the end? And how large were the IOs? I know... Ishould just try myself. But if you have it handly ( sorry :-), then please share here or in an Email ).


Hope this helps,
Hein van den Heuvel
HvdH Performance Consulting

[Rajeev jain]

Hein,
Your point would be valid if I see r/wr time same on local drives and SAN drives. They are 3 times apart. I dont think the file cache within the OS distinguish between local drives and SAN drives, right?

What make me believe here is that either the paths to the disk are saturated which could be a driver issue OR an issue the cards itself.

Rajeev

[Hein van den Heuvel]

Ah! Good work.
I missed a step in reading your details.
Thanks for making it explicit.

>> Your point would be valid if I see r/wr time same on local drives and SAN drives. They are 3 times apart. I dont think the file cache within the OS distinguish between local drives and SAN drives, right?

Absolutely Correct.
The cache does not discriminate
(except possibly for RAM disks, and explicitly selected mount options: http://g4u0419c.houston.hp.com/en/B3921-90010/mount_vxfs.1M.html )

>> What make me believe here is that either the paths to the disk are saturated which could be a driver issue OR an issue the cards itself.

I would have to agree and would lean back towards something, potentially hardware, being broken or mis configured.

How about the hp-ux scsi queue_depth?

Any counters on a SAN switch that might help you?
Any counters on the storage controlled that might help you? Notably queue depth?

Any error/retry accumulators growing?

Hein.