HPUX 11.11 I/O tuning.

[Royal]

We are having some timing issues running the following command "cat | gunzip | tar xvf". The input file is 17.7GB. This works but it seems that the process isn't running the best that it can. We run 8 stings of this each writing to it's own file systems, vxfs, on external disk. The system has 21GB of memory. We had 6 processors on this system, 1 was added for a total of 7, by adding the 1 processor it took longer for these run. (about 20 minutes longer) Our dbc_min/max_pct is currently set to 8/10, if I increase these to say 15/20, would that help my performance? Are there other parms that could be changed that would help. "sar -b 5 5" shows, 07:43:06 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
07:43:11 507 120642 100 8958 136577 93 0 638
07:43:16 508 121214 100 8981 137255 93 0 595
07:43:21 399 120108 100 8975 136061 93 0 607
07:43:26 408 122670 100 9106 139106 93 0 727
07:43:31 587 121824 100 9031 137939 93 0 640

Average 482 121290 100 9010 137386 93 0 642

[Steven E. Protter]

Shalom,

There are limits to what you can accomplish here.

http://my.safaribooksonline.com/0131433490/ch06lev1sec6

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1251957

If you are demanding to much of your system, you will get these kind of results.


SEP

[Hein van den Heuvel]

How would you like to have a cache help here?
Think about it for a moment !

(hard as that might be for a person who can not figure out to set up a proper 'author' name in the forum)

A lot of gzipped data is read once, never to look back at.
And a lot of un-tarred data is written never to be looked again.
How is more memory for caching going to help other then making the system waste more cycles trying to remember more, but no longer useful, stuff?

Why was a CPU added? What was the justification? Had one lying about? an expensive hunch? Serious performance indicators?... share those!

For serious help please specify OS version and platform and perhaps an indication of the IO system capabilities, and consider the courtesy of signing of with a real name.

(make one up if you have to. It will make us feel better trying to help a real person with a name).

Best regards,
Hein van den Heuvel

[Royal]

My bad, I'm Royal Hagens... I thought that was the title to the question...

A CPU was taken from another LPAR, added in hopes that it would help, it didn't and it will be reassigned to the other system tonight.

How would you like to have a cache help here?
Think about it for a moment !

The additional Cache, we thought would help with the "cat" being piped to the GZIP being piped to the tar.

The system is a hpux 11.11.

Thank you for the JFS Tuning and Performance document, I'll need to read that in more detail, there appears to be something in here that may help us.
This is a training system for one of our applications, normally the system usage is low, except during refresh.

[Dennis Handly]

>I'm Royal Hagens. I thought that was the title to the question.

(I'm wondering why? You use the same profile name for 100s of questions.

You can change this by going to "My profile" link in the top left. Then to "view or modify forums profile", then "Forums user name:".)

Any reason you are using "cat x.gz | gunzip"?
You only need this if your x.gz file is over 2 Gb. (Ah, you said 17 Gb!) If not, using the following would eliminate the pipe bottleneck:
gzcat x.gz | tar -xvf -

You may consider downloading the gzip sources and making your own changes to support large files. I don't know if the gnu tar will handle large gzipped files?

>The additional cache, we thought would help with the "cat" being piped to the GZIP being piped to the tar.

The cache is wasted for pipelines. There is only a 8 Kb buffer and you have two of them.

[Hein van den Heuvel]

>> My bad, I'm Royal Hagens... I thought that was the title to the question...

No harm done.
As that name seems relatively unique, in the restricted context of hpux interest, you appear to have create a profile in the past:
http://forums11.itrc.hp.com/service/forums/publicProfile.do?userId=CA38527&forumId=1


>> A CPU was taken from another LPAR, added in hopes that it would help, it didn't and it will be reassigned to the other system tonight.

That's a little what those partitions are for, to be able to do these experiment without major undertakings. Thanks for clarifying!

>> The additional Cache, we thought would help with the "cat" being piped to the GZIP being piped to the tar.

Understood, but cat would not read the whiole file. The pipe will block its progress, so there are still only a handfull kilobytes of immediate interest. Everything else is distant past or unknown future.

fwiw... sometimes you can help looking into the future by pre-reading from a different stream, but that's hard/tricky to set up reliably so. And the file system read-ahead will only carry you so far.

>> The system is a hpux 11.11.

Thanks.
11.31 is supposed to be smarter with the buffer cache.

Regards,
Hein.

[Royal]

Thanks to all for the responses. The "gzcat" option is a good one, I'm running some test on that to get some timings.
There was a link to JFS Tuning which I refered too earlier, it was in a link to another response. Our current mount options are very vague, we only "vxfs delaylog 0 2" in /etc/fstab, quickly looking at the JFS Tuning document, we might want to add some other options, datainlog for one, I'm not to sure about the others, still attempting to read though it. If there are some suggestions that would be nice.

Now, I don't own the process that is having the issues, "cat | gzip|tar", this is a very old process that another group owns, we're just trying to be helpful.
Thanks to all, again.

[Dennis Handly]

>The "gzcat" option is a good one, I'm running some test on that to get some timings.

Sorry, that won't work for over 2 Gb unless you have a newer? version of gzip/gunzip/gzcat.