Disk bottleneck question

[Sean OB_1]

Hello.

If I have a filesystem that has the primary over a lun with 6 disks and has it mirrored to another array, also on a lun with 6 disks, how much impact does the mirror have on disk I/O?

If the mirror is slower than the primary will that hold up the application, or will mirrordisk simply cache the mirror writes and write them when it can?

I want to add 6 disks to the primary lun and need to know if I should also add 6 disks to the mirror or simply replace the existing 18gb drives with 36gb to have the same capacity.

TIA,

Sean

[George Petrides_1]

Sean,
Is this hardware mirroring in the array or software mirroring with Mirror-UX. The first way depends on the manufacturer. Mirror-UX makes two writes since every logical extend in the logical volume has two physical extends on two different disks. I have not seen big performance impact from mirroring in terms of writing, however, in cases where writing was heavy then striping (extend based) helped a lot. However, reading is faster with Mirror-UX since it can simultaneously read from two different disks. Now, regarding waiting for I/O to mirrors, in the case of EMC or XP for example, the server does not know anything about mirroring, however, with Mirror-UX unless you changes the write mode to asynchronous then the machine does not proceed to the next I/O until it receives confirmation from the I/O subsystem. The default mode is synchronous and you can change it in the kernel, however, I wouldn't do that cause it might cause big problems in the event the machine crashes and OS and SW thing that data has been written to the disk (did not wait for confirmation) but it really wasn't.
Hope this helps,
George

[Sean OB_1]

Sorry I should have given more detail.

The primary is on an FC60, one lun with 6 disks raid 5 5+1

The mirror is on two sc10's, one lun with 6 disks.

The mirroring is via mirrordisk ux.

[S.K. Chan]

If you're talking about LVM mirroring and have concern about it's performance .. don't be.. in fact it may improve the "read" performance. What happens is for a read operation, it first looks at the primary disk to determine if the IO queue is ok or not, if ok it'll proceed to read from the primary disk, if not ok (for example IO queue is congested), it'll read from the mirror disk instead. Hence the read operation is very efficient. For write operations, they are performed in parallel, so it does not makes any difference (single disk vs mirror disks).

[Sean OB_1]

Let me give you some background on this.

This is a unidata database application. The primary data is written to a volume that resides on an FC60 array and is on a 6 disk lun (raid 5 5+1).

This is then mirrored via mirrordisk to a couple of sc10's in a different building (6 disks also).

We are seeing very high disk utilization and I/O wait times are unacceptable. Both the primary and mirror "disks" are virtually always above 90% utilized when checked via sar -d.

In addition we are running out of space so we need to increase it anyway.

On the primary we will be moving data off the fc60 to a different array, which will free up 12 drives for our use. I want to either create a second lun with those 12 drives and move some of the production data to the new lun. This would spread some of the i/o over 18 disks instead of 6 now.

On the mirror I'm more limited on disk bays. So I have the option of filling the remaining bays with 12 more drives giving it 18 as the primary has, or doubling the size of the current drives and using 9 drives instead.

My concern is that the spindles themselves are the bottleneck and if I only have 9 on the mirror it will defeat the increase of drives on the primary.

So I guess the question is this. On the mirror will I see a performance hit if I only use 9 drives instead of 18?

[George Petrides]

OK, never worked with Unidata but have lots of experience with Universe that I think is very similar. Anyway, having 100% utilization is no problem, the problem arises if the queue to the disk is high. Think about bank tellers, if the tellers are 90% of the time busy who cares, if the line of customers is long then that's a problem. So, when you use sar look at the next columns that is the queue of the disk requests. When it comes to the SC10 JBOTS, you might want to stripe those but I never had a situation where only the mirror was striped. You might either want to stripe your data on the FC60 and SC10 or create the initial volume on the SC10 with manual striping (extend 4MB with a script) and then mirror on the SC10 or if don't want to delete and restore data you can create a striped volume on the SC10s (exact sizes as the volumes on the FC60) and then stripe.
Sean, you might also want to start assigning points since it will help you get more answers from more people.
Cheers,
George

[Donald Kok]

Hi Sean,

I would make sure yhat the disks in each lun are striped, since the I/O is handled in paralel. This works much faster (both read and write) than just 6 disks. MirrorUX over these 2 luns makes sure your data is redundant. It also improves the reading.

Hope this helps
Donald

[Keith Clark · Accepted solution]

Hi Sean,

Just remember when using software mirroring your bottleneck is always the slowest piece of your storage subsystem. Read performance on mirrors is often enhanced by the mirroring software using either disk for read (whichever is available to service the request). Write performance is always impaired. When the O/S issues a write, it is not acknowledged as complete until both sides of the mirror respond. If you mix different speeds of disks and longer distances, the slowest combination is your bottleneck. In your case this is almost certainly the sc10's in the other building.

A quick test? Collect some I/O stats, then break your mirrors and repeat. I think you will find the sc10's in the other building are slowing you down.

Also look at the queues as George pointed out. High utilization is good, longs queues bad.

For performance it is always best to use the same components. Same disks, same geometry and same interfaces. Also ensure that you spread the I/O over as many hba's and spindles as possible.

Good luck,

Keith

[Sean OB_1]

George,

as my profile shows, I give points for every answer I get. I'll get to it, but not until I know the answer and can determine what points to give to whom.

But since you were so kind to chastise me I'll give you your zero points now. :-P

Sean

[david gao]

Lots of above replies give very good suggestions and explanation with mirro, stripe and even asyn I/O.

obviously, channel to SC10 is slower than FC60. It's better to use the same diskarray.

One more suggestion is turn dynamic cache buffer on.