topic Re: Cache blocked in Operating System - HP-UX

Cache blocked

Shahul — Wed, 27 Nov 2002 17:10:06 GMT

Hi

I have a problem, Hope some one from U will be able to give quick solution. In my N class box, we have oracle loaded. I have a weekly script which will shutdown that database and copy data from one filesystem to another filesystem for backup purpose. This script was getting completed with in 4 hours earlier. Now all of a sudden script is taking hell lot of time. The command I am using for copying is

tar cvf - ./* | tar xvf - //.

When I go to glance and check for details, I could see 90% is blocked on cache. Anyone has any suggession?

The only change we have done is in array. Some stripped Disks we change to Raid 5. But it is nothing to do these particular filesystems or VGs.

Thanks in advance
Shahu

Re: Cache blocked

Darrell Allen — Wed, 27 Nov 2002 17:23:52 GMT

Do the disks your tar'ing to and from share the same i/o chain as the disks that are now raid5? How's performance for the raid disks since you changed?

One thing that will help is to use cpio instead of tar. It's much quicker for disk to disk copies:

$ cd /source_dir
$ find . | cpio -pudlmv /destination_dir
omit ???v??? (verbose) option from cpio for more speed

Darrell

Re: Cache blocked

Justo Exposito — Wed, 27 Nov 2002 17:25:36 GMT

Hi,

Do you change dbc_max_pct parm in your kernel recently?

Regards,

Justo.

Re: Cache blocked

Shahul — Wed, 27 Nov 2002 17:38:55 GMT

Hi

Thanks for the replies. See, I have done changes in some other VGs. Not in this. These were RAID 5 earlier and now. That's what I mentioned it is nothing to do with this FS. Anyway my source VG is this

/dev/vg07

-------------------

/dev/dsk/c4t0d2

/dev/dsk/c15t0d3

/dev/dsk/c4t0d4

/dev/dsk/c15t0d5

/dev/dsk/c4t0d6

/dev/dsk/c15t0d7

/dev/dsk/c4t1d0

/dev/dsk/c6t1d1

/dev/dsk/c16t1d2

/dev/dsk/c6t1d3

/dev/dsk/c16t1d4

/dev/dsk/c6t1d5

/dev/dsk/c16t1d6

/dev/dsk/c6t1d7

/dev/dsk/c16t2d0

/dev/dsk/c15t0d2

/dev/dsk/c4t0d3

/dev/dsk/c15t0d4

/dev/dsk/c4t0d5

/dev/dsk/c15t0d6

/dev/dsk/c4t0d7

/dev/dsk/c15t1d0

/dev/dsk/c16t1d1

/dev/dsk/c6t1d2

/dev/dsk/c16t1d3

/dev/dsk/c6t1d4

/dev/dsk/c16t1d5

/dev/dsk/c6t1d6

/dev/dsk/c16t1d7

/dev/dsk/c6t2d0

/dev/dsk/c15t2d1

/dev/dsk/c4t2d1

/dev/dsk/c4t2d2

/dev/dsk/c15t2d3

/dev/dsk/c4t2d4

/dev/dsk/c15t2d5

/dev/dsk/c4t2d6

/dev/dsk/c15t2d7

/dev/dsk/c4t3d0

/dev/dsk/c16t3d1

/dev/dsk/c6t3d2

/dev/dsk/c16t3d3

/dev/dsk/c6t3d4

/dev/dsk/c16t3d5

/dev/dsk/c6t3d6

/dev/dsk/c16t3d7

/dev/dsk/c15t2d2

/dev/dsk/c4t2d3

/dev/dsk/c15t2d4

/dev/dsk/c4t2d5

/dev/dsk/c15t2d6

/dev/dsk/c15t3d0

/dev/dsk/c6t3d1

/dev/dsk/c16t3d2

/dev/dsk/c6t3d3

/dev/dsk/c16t3d4

/dev/dsk/c6t3d5

/dev/dsk/c16t3d6

/dev/dsk/c6t3d7

/dev/dsk/c4t2d7

And Destination VG is

/dev/vg08

B:&^

/dev/dsk/c6t7d7

/dev/dsk/c16t7d7

This output is from /etc/lvmtab.

Then kernel parameter, I have not changed any kernel parameter recently.

I hope this will help U.

Rgds
Shahul

Re: Cache blocked

John Poff — Wed, 27 Nov 2002 18:19:05 GMT

Hi,

What version of HP-UX are you running? What are the mount options on the filesystems? Are you using 'mincache=direct'? How much buffer cache do you have configured?

JP

Re: Cache blocked

Oleg Zieaev_1 — Wed, 27 Nov 2002 19:01:31 GMT

Hello Shahu.

I would be more concerned about frame performance. You have introduced Parity volumes to your stripe set. You may concider revisiting your stripe size to align it with average IO size you have on your volumes.
Crossing the same IO channels and the same spindels may also inject performance bottlenecks. In case you have few partitions on one physical drive, it will impact performance of the whole disk and eventually - performance of the partition you are writing to.
Small writes on RAID5 might have performance impact as you still be writing the whole stripe set, effectively increasing 'busy' timeouts for the other partitions. Performance will suffer even more, as parity have to be recalculated even if you change a bit.
So, my suggestion would be to check layout for your partitions on the array and optimize them.
How do you copy information (duplicate), i.e. tar or cpio will not drastically improve your performance.

Increasing dbc_max_pct might help you as mentioned above as well as a mount options.

I would also check with DB vendor if you can use fs_async.

Hope this helps,
0leg

Re: Cache blocked

Shahul — Wed, 27 Nov 2002 19:10:05 GMT

Hi

Thanks

Now I am trying with cpio. Seems little difference. But not as expected. dbc_max parameter is 15 and min is 5. Anyway I have not changed those. Apartfrom that an outage is a dream. Blocked on cache is less in cpio compared to tar. Now for 3.5Gb it is taking 17 mts, Is it reasonable? I think it can much faster than this since it is in same system and in different channels.

Thanks in advance

Shahul

Re: Cache blocked

S.K. Chan — Wed, 27 Nov 2002 21:11:21 GMT

I agree with Darrell about using cpio instead of tar (for disk to disk). The throughput you're getting is apparently pretty good, at a rate of about 210MB/min. I tested on my local (standalone) drives (ultrascsi) and I'm getting something like 35MB for 20sec of cpio which works out to about 105MB/min. Down the road you may want to investigate a more efficient tool for frequent data files synchronization. Here we use "rsync" which does a better job that any of the native tools.

Re: Cache blocked

Shahul — Fri, 29 Nov 2002 10:26:38 GMT

Hi

Thanks to all. I have acheived little more improvement by recreating the whole target filesystem. Because I found that the block size of target filesystem was lesser than source. But I am still hungry for speed...

Searching for it..

Thanks in advance
Shahul