help an app. is beating up my disk!!

[someone_4]

Hey everyone,

Our trouble ticket software is killing my disk on the server. I have narrowed it down to to one application because when I stopped/killed it everything went back to normal.

I started out with a memory issue so I made dbc_max_pct 10 since I have 2048 MB. And now my disk is getting the crap beat out of it by this app.

I have attached my glance output here.

According to the docs I have read I need to distribute the app into varius disks so that the I/O is aslo distrubuted.
The app has its own lvol on disk c1t2d0 and I have another disk that I can use.

So in order to let this app use the other disk would I umount it extend the lvol and let it take a peice from the other disk?

~ Richard

[Justo Exposito]

Hi Richard,

I have the same memory like you and when I try to reduce the dbc_max_pct to 10 I have a pour throughput in the omniback backup, then I must to put 25 in the dbc_max_pct again.

Regards,

Justo.

[Sandip Ghosh]

Extending the LV will not solve your problem. Take the backup of the existing filesystem, and create a new filesystem stripped on both the disk. Then only it will share the load of the disk.

Sandip

[Sandip Ghosh]

Regarding the performance of dbc_max_pct you have to look at sar -b. Average %rcache should not be below 90%. It directly depends on dbc_max_pct.

Sandip

[RAC_1]

the effect of dbc_max_pct is to checked with sar -b if rcache rate is bellow 90 or so you can think of tuing that value.

Whet you can do is add this other disk to volume group and recreate the lvol with distributed option.

(backup everything that will be required for restore)

[A. Clay Stephenson · Accepted solution]

Well your plan would work but it would probably offer no significant improvement. The second disk would only be used when all the PE's in the first disk were used. Whay you really want to do is stripe that LVOL across both disks so that sucessive chunks come from alternating disks.

You woule need to backup this filesystem and then remove the LVOL using lvremove (after unmounting the filesystem). You would then recreate the LVOL using lvcreate -i 2 -I 64 ... to stripe across two disks in 64K chunks. Run a newfs and then restore from backup.

Ideally, these disks would be on separate controllers and you might want to stripe across more than two disks.

Now having said this, it appears that the real key for improvement lies in the application itself. You need to contact the developers and tell them what you are seeing. If the data you posted is the typical, all-day long norm then you are hammering the disks and there may be a better way.

[steven Burgess_2]

A.Clay

What if the lvol was in an array, would the same theory apply ?

Steve

[Paula J Frazer-Campbell]

Richard

Do you have online jfs? If so try a defrag first.

If no online jfs then backup your data (twice) rm the data and restore.

Before you go into a disk restructure prove what is at fault.

Pick up the PIDs of the app and using Glance and Tusc find out which bit is either doing big searchs/reads/writes.

Too many time we restructure / use more cpu / memory to make crappy apps work.


Paula

[A. Clay Stephenson]

If this were an array LVOL and the array had lots of cache and the performance seemed to be okay then I wouldn't worry about it.

Glance has no way of knowing that this is not a simple disk; all it knows is that a lkot of I/O is going through one device.

However, even arrays can often benefit from striping in the sense that the data is split across multiple I/O paths. The actual bottleneck may be in the host so that sending I/O through multiple SCSI controllers improves overall throughput even if the array itself is not the bottleneck.

In my earlier response, I almost added that if Richard can't get the application code improved then a real option might be a solid-state disk.

In that case, Glance would still think that the disk is over-utilized eventhough the actual performance might be stellar. You have to know what the numbers mean.

[Anil C. Sedha]

Richard,

Let me get some steps clear

Your application was utilizing a lot of i/o or memory from the system ??

If it was memory, then you need to work on the swap issues. But i believe your application is creating problems on the I/O end. If that's the case, then verify what disk it is on, see the difference in the spare disk that you have (size & type of disk)

You may extend the application on the other disk, but if the other disk is bigger in size than the current one, I would suggest doing a "pvmove" from your current disk to the new disk. It is just a 3-4 steps thing and is very simple.

I faced this issue once and it got resolved.

Regards,
Anil