cp command in 11.11 runs slower and slower

[Gerard Drobnicki]

Has anyone experienced slower cp times in 11.11 versus 11.00 ?
We recently upgaded an N-class running 11.00 to an rp7400 running 11.11 with more horsepower and better fibre cards. However, a hot backup job has actually degraded from 3 hours to 4 hours. This job hot copies 243 oradata files [75 Gb] from an EMC Symmetrix to an HP Model 30 Array. Under 11.00 this job took 3 hours. After the upgrade, the first night it ran in 2 hours and 45 minutes, but then steadily deteriorated each night until it settles down at 4 hours. We've tried the following changes:
(1) lowered dbc_max_pct from 50 to 10
(2) increased disksort_seconds from 0 to 4
(3) upraded fc driver to B.11.09
(4) rebooting just resets the problem
Any advice would be appreciated.

Thank,
Gerry Drobnicki
Systems Admin
IPaper

[S.K. Chan]

The only thing I can think of is 2 things ..
1) Your buffer cache, setting it too low might increase io on disks, hence cp is slowed down, but that doesn't explain the "deteriorating" cp performance.
2) Explore "vxtunefs" command to tune your vxfs filesystem. On 11i there is a parameter in vxtunefs called "max_diskq". I'm going to just cut-&-paste the section of "man vxtunefs" here ..

max_diskq
Limits the maximum disk queue generated by a single file. When the file system is flushing data for a file and the number of pages being flushed exceeds max_diskq, processes block until the amount of data being flushed decreases. Although this does not limit the actual disk queue, it prevents synchronizing processes from making the system unresponsive. The default value is 1 megabyte.

I seriously think increasing the default 1MB would help your situation. I have never try this before but I think the vxtunefs would probably be like so ..

# vxtunefs -s -o max_diskq= /

[Pete Randall]

Gerard,

If I read your post correctly, re-booting brings the run time back down to 2.75 hours. Is that true?

Like S.K., my first inclination is to suspect the memory subsystem - but the fact that it deteriorates gradually over subsequent runs kind of negates that approach unless it's some sort of leak.

I would pursue this with HP support to see if there's a patch that would help.

Good luck,
Pete

[Gerard Drobnicki]

S.K. => Our filesystems that contain oradata files on EMC disks are all mounted with mincache=direct. On the Model-30 only read cache is enabled. Does this affect your suggestion? Thanks.....

Pete => With the kernel changes I've made so far, a re-boot doesn't even get the job under 3 hours now - it resets to 3 hours & 15 minutes - then degrades back up to 4 hours.

[Uday_S_Ankolekar]

Also look for any memory leakage.

Try running sar -v 10 20 and see if you have any overflow.
May be increase of nfile would help inorder to do this increase maxusers (if you are using the sam formula). This requires a reboot.

-USA..

[Pete Randall]

Gerard,

A search of the patch database for +cp +slow came up with PHNE_25625 ONC/NFS General Release/Performance patch. Unfortunately, it's no longer available so I can't even look at the defect description to see whether it was a good match or not. I'm still inclined to think it's a patch issue - just not sure where to go from here.

Pete

[Gerard Drobnicki]

USA => There are no memory leaks; however, 11.11 has a sar -v bug & displays all inod-sz as 0/.....

[Gerard Drobnicki]

Pete => we have PHNE_25625 installed & we have no other problems with any online transactions, reports, or jobs. Just this backup. HP had pointed me to the kernel parameters, but disksort_seconds seems like trial & error: I've tried the default of zero and now 4.....

[A. Clay Stephenson]

Hi:

While I tend to run 10.20 and 11.0 Oracle boxes with the vxfs option mincache=direct,convosync=direct,delaylog,nodatainlog, I have found that on 11.11 boxes, using the traditional mount options delaylog,nodatainlog
and thus utilizing the buffer cache is the better performer - especially on machines with fairly large amount of memory. Also, the boxes benefit from somewhat larger buffers caches (around 700-800 MB or so).

I would turn off dynamic buffer cache by setting bufpages to about 204800 (800MB), enable the filesystem buffers by simply mounting delaylog,nodatainlog.

I think that if you do this you are going to see better performance in Oracle and certainly much better performance in your copy operations.

[S.K. Chan]

Clay's suggestion is something you can try implement immediately to see if it helps as it was quoted from his own experience, nothing can beat that.
Also take a look at "Performance and Tuning" section in ..
http://docs.hp.com/hpux/onlinedocs/B3929-90011/B3929-90011.html
before you dive into vxfs tuning.

[Gerard Drobnicki]

Clay => I think we're going to try your suggestions. If we set BUFPAGES to 204800 would we have to set NBUF to half of that?
Thanks.....

[S.K. Chan]

If you don't mind .. yes

bufpages (4096-byte memory pages)
nbuf=bufpages/2

[A. Clay Stephenson]

No, I suggest that you set nbuf to zero to enable the default behavior of bufpages/2 buffer headers allocated. (Which in your case was where nbuf was going to be anyway - the bufpages/2 value (by setting nbuf to zero) is a rather good value.

[Gerard Drobnicki]

Well, in a nutshell, last night we probably changed too much at once:
increased bufpages from 0 204800, left nbuf at zero, removed mincache=direct, changed detainlog to nodetainlog, and reduced disksort_seconds from 4 to 1. Also EMC upgraded the microcode on our Symmetrix from 52.65 to 52.66. Unfortunately, our original 3-hour backup still ran in 4 hours and Oracle Apps are running slow right now. Any thoughts would be appeciated. Thanks.....

[A. Clay Stephenson]

You certainly changed many things at once. I have made the assumption that this new box has
plenty of memory and is not paging out.

There is something you need to check right away (and this would explain everything): Look at the value of timeslice - there has been a serious flaw in one of the tuned paramers sets for databases that incorrectly set timeslice to 1 rather than 10. With a timeslice of 1, as the machine gets loaded it begins to do context switching and very little else. If your timeslice is set to 1, set it to 10 and I assure you that your situation will dramitically improve.

If timeslice is ok then it is really time to pull out tools like Glance and find where the bottlenecks are.

[Gerard Drobnicki]

Clay, timeslice is set to 10 and glance plus alarms are clear - occasionally it reports disk bottlenecks. I think I'm going to have to go back to disksort_seconds = 0 and work from there.....

[A. Clay Stephenson]

I've found a pair of VxFS patches that look quite promising. PHKL_26230 & PHCO_26252.

You haven't mentioned what the mount options are for the filesystems on the Model30.

[Gerard Drobnicki]

You're right Clay, the 2 patches sound very promising! For tonight though, I've already used vxtunefs [as per H.K. originally] to bump up max_diskq from 1m to 4m. Also, max_buf_data_size from 8k to 64k and max_direct_iosz from 1m to 2m. All mount options for EMC drives & the Model-30 were changed yesterday from delaylog,detainlog to delaylog,nodetainlog. [Was I wrong with this?] Also, mincache=direct was removed for all drives containing oradata because we went with your new os buffering. The Model-30 is just one big 120 gig filesystem with the same options. Thanks....

[A. Clay Stephenson]

No that's fine. I just wanted to make certain that the destination filesystem was also using buffer cache.

[Gerard Drobnicki]

Using S.K. Chan's vxtunefs suggestion we bumped up max_diskq from 1m to 4m. We also increased max_buf_data_size from 8k to 64k and max_direct_iosz from 1m to 2m [it wouldn't take 4m]. This resulted in reducing the previously-3-hour-now-4-hour backup by 20 minutes - a huge step in the right direction.
I think this supports Clay's findings in PHKL_26230 where it fixes "behavior that can cause VxFS 3.3 file system performance problems when sequential I/O requests of less than 64KB are performed. This behavior can affect backup utilities and other applications that perform sequential I/O accesses to the file system" and in PHCO_26252. We will schedule these 2 patches to go in very soon. Also, I think we will set disksort_seconds back to 1. One thing I had always taken for granted on the 10.20 & 11.00 boxes was always mounting delaylog,detainlog,mincache=direct. What Clay pointed out, using the 11.11 os caching and mounting delaylog,nodetainlog - well it's a whole new tuning world right now. Thanks.....

[Gerard Drobnicki]

After setting disksort_seconds back to zero the Big Prod Backup runs in 3:30 [still 1/2 hour slower than expected, but faster than 4 hours & holding steady.]

[Bill Hassell]

If these files are fairly large (dozens to hundreds of megs), I would suggest using cpio -p to copy the files. cp is very inefficient for large files. Use something like this to copy the files:

Store the names of the files to be copied into a file. Then:

cat file_name | cpio -pudlmv /destination

cpio -p uses very large buffers that are optimized for disk-to-disk transfers.

[Gerard Drobnicki]

Bill, there ares 243 oradata files [75 gig] and I don't want to copy the entire path. Can I still use cpio ? Thanks.

[Bill Hassell]

Since you already have some sort of luist of the needed files, either file by file or by using a filename match, you can pipe the names of these files into cpio. For example:

cd /data01
print d*ab | cpio -pudlmv /destination_dir
print c*00 | cpio -pudlmv /destination_dir

and so on. NOTE: cpio does not handle large files (anything over 2Gb) so be sure to check on this limitation. If you have a 100-200 meg file, you could compare the speed of cp and cpio for copying the same file.

[Gerard Drobnicki]

Thanks Bill, we'll give it a shot.....