Mixing differently sized disks in a mirror

[Ralph Grothe]

Friday night I had to allocate new disks for a D370, which had been added together with a HVD10 disk enclosure.
Copying and mirroring of some 30GB took quite a while, but had to be done, as the old VG containing the LV which needed to be extended had to be desolved altogether.
This was necessary because at the time when the VG was created (by whom ever) it used to consist entirely of 4GB disks (must have been a big size then).
Those who created the VG obviously didn't care to feed vgcreate with the -e flag, with the consequence of only allowing allocation of 1023 PEs of any PV joining the VG.
Because I had 4 new 18GB disks I used two of them to create a new VG (now with -e 10000, i.e. max PEs/PV).
This new VG I needed two copy the 30GB of data from the old VG which I had to desolve.
Because I had to spread one LV over both 18GB PVs I couldn't use -C y (i.e. contiguous allocation).
After the data had been transferred, I removed the old VG and forced a new volume header on the freed 4GB PVs by pvcreate -f /dev/rdsk/c?t?d?.
I then extended the new VG (of the two new 18GB disks) by those 4GB PVs.
I decided to do so (instead of mirroring against two other new 18GB disks, which would have matched in size exactly) because all new disks in the new disk enclosure are connected to the same SCSI bus, SCSI controller, and power supply.
Unfortunately I couldn't avoid some PEs of the 18GB PVs and its mirror PEs on the last 4GB PV to be connected to the same bus/controller.
So maybe I haven't gained higher redundancy at all?
Now I am very doubtfull if it wouldn't have been more sensible to mirror against the two other 18GB disks of same size. make, and type, despite the fact that all of them are on the same controller.
Is it sensible to mix different size disks in an LVM mirror?
Ah, I forgot to mention that the actual building of the mirror (by lvextend -m 1) took hours.
This may have been owe to leaving the default PE size to 4 MB.
I think I should have used the -s flag and assign 8 MB.
Because I have to do the same with another D370 this night I think I can do better with some hints from you.

Finally one question which came up my mind when I edited /etc/fstab to update the mount record of the new VG and LV.
The sixth entry in this file accounts for the order in which fsck should be executed.
Since the only left hfs filesystem on our boxes is /stand and all other local filesystems are vxfs the Veritas journaling filesystem, I cannot see any use or relevance of this entry for journaling filesystems.
Am I wrong in this assumption?

[Roger Baptiste]

<>

I didn't understand this part. You said
the OLD VG had two 4Gb disks . So how
come you had to copy 30Gb data??

<>

A very common error. Was this VG00??


<>

Ok. So, you now had the OLD VG (of 4Gb disks) and a new VG of two 18Gb disks
and other two 18Gb disks not used at this
point of time.

<>

What was the size of the largest LV on
this old vg?? Was there any LV more than
18GB?? If that was the case, then you could
not obviously use the two 18Gb's disks for
contigouos allocation.

<>

I think it could done, unless you have
a LV which is above 18gb.

<>

Ok.

<>

I think a better option would have been
to keep the 4GB's in a separate VG, rather
than mixing it with the 18Gb disks.

<>

That is true, but you would still get
Disk Redundancy by mirroring the 18Gb disks!
In that way you protect yourself against
disk failures. Right now, you have no such
protection, unless the unit has a builtin
sparedisk/failover option, like some
of the small Raid units do.
The factor of whether to mirror or not is
more dependent on the need for disk space.

<>

So you mirrored the 18Gb's to the 4Gb's??
Not sure, whether it is a good idea.



Nope.

<>

Yes, the better option IMO is to
club the 18Gb disks in one VG with mirroring;
and clubbing the 4Gb disks in another Vg
with mirroring. You will not get controller
redundancy, but atleast you will protect
yourself against disk failures.

<>

Not advisable, unless there is a crunch
in disk space.

<>

Not surprising with 30gb of data and
different disk sizes being mirrored ;-)

<< I cannot see any use or relevance of this entry for journaling filesystems.
Am I wrong in this assumption? >>

Well, i think it still does the same job of setting the order on how to mount the filesystems. In database systems, i give the priority of db volumes lesser (higher value) than the other filesystems. But, one
can get along without it.

HTH
raj

[Thierry Poels_1]

Hi,

mirroring the 18G on the other 18G would be easier in administration. But, as you mention, those disks are on the same controller = speed drops down and if controller is gone, disk are gone.
There is nothing wrong in mirroring large disk on several smaller ones. But I would user Physical Volume Groups to group the disks within the volume group (/etc/lvmpvg) and use PVG-strict allocation policy (lvchange/lvcreate -s g) to prevent a mixup of disks.

good luck,
Thierry.

[James R. Ferguson]

Hi Ralph:

Several comments. My personal choice is to mirror logical volumes from and to similar size disk. In that way, it is more likely that if I 'lvextend' a logical volume, I will have the number of extents I need on the mirror, given that the first rule is not to allow mirror copies of a logical extent to share the same physical volume. I find this makes long-term maintenance much easier particularly when a volume group consists of several mirrored logical volumes.

You might consider using 'pvmove' or simply rebuilding your mirrors if you agree with this. I cannot see how your choice of a 4MB default PE size impeded the speed at which mirrored extents were replicated. I don't think a larger default size would necessary improve the mirroring rate.

As regards mirroring logical volumes on physical volumes that share the same controllers, I find that sometimes that can't be avoided for other choices of disk geometry. Ideally for high-availability and for throughput, you want to mirror using different controllers. However, the chance of a controller failure is far less than the chance of a disk failure, so if my only choice is to mirror with the same controller, as opposed to not mirror, then to me the choice is obviously to mirror!

Lastly, I don't really think editing /etc/fstab to modify the pass number for file system checks matters too much. I always use the default values (1 for the root filesystem and 2 for all others).

Regards!

...JRF...

[Carsten Krege · Accepted solution]

> Unfortunately I couldn't avoid some PEs of
> the 18GB PVs and its mirror PEs on the last
> 4GB PV to be connected to the same
> bus/controller.
> So maybe I haven't gained higher redundancy
> at all?

The protection against disk failures is the same as if you used the 18GB disks. It doesn't make a difference if you use the 18GB or 4GB disks.
If the SCSI card/cables fail where mirror and master are configured on, all your filesystem will be corrupted who have master and mirror extents on this SCSI bus.

lvols that have mirror and master on separate SCSI busses will continue to operate. You only added further redundancy to lvols who have well separated master and mirror PEs.

> Now I am very doubtfull if it wouldn't have
> been more sensible to mirror against the two
> other 18GB disks of same size. make, and
> type, despite the fact that all of them are
> on the same controller.

In the worst case you have the same redundancy.

> Is it sensible to mix different size disks
> in an LVM mirror?

It doesn't make a difference from an LVM point of view.

> Ah, I forgot to mention that the actual
> building of the mirror (by lvextend -m 1)
> took hours.
> This may have been owe to leaving the
> default PE size to 4 MB.

No, increasing the PE size won't have any improvement on the sync performance. I would assume that a disk is able to do ~10MB/s sequential write. For 36GB you would therefore need (36*1024)GB / 10MB/s = 3700s = ~1h.

This result really depends from the block size used though. For tiny block sizes of 1-2k the results are much worse, for sizes of 64k and up this result should be ok.
The PE size doesn't reflect the internal size LVM uses to do its write requests.

You might want still use a bigger PE size, since the LVM header's size is limited. Since you need an entry for every PE in the LVM header you might run out of space (file too large message).

Finally one question which came up my mind when I edited /etc/fstab to update the mount record of the new VG and LV.
> The sixth entry in this file accounts for
> the order in which fsck should be executed.
> Since the only left hfs filesystem on our
> boxes is /stand and all other local
> filesystems are vxfs the Veritas journaling
> filesystem, I cannot see any use or
> relevance of this entry for journaling
> filesystems.
> Am I wrong in this assumption?

The passnumber is still relevant for VxFS filesystems. You should ensure that the "/" and /stand filesystems have the passnumber 1 and the rest has 2 or higher. Since VXFS is usually (but not always) very fast in fsck'ing the filesystems, the importance is not that high as it used to be with HFS. However, if there is a filesystem /garbage that has a flag in the superblock set that a full (hfs like) filesystem check is required, you probably don't want it to be checked before all other important filesystems were completed.

Carsten

[harry d brown jr]

Mirroring ???UNLIKE??? disks is not recommended because it reduces the performance of mirroring dramatically. Performance of mirroring will be based upon disk size, rotation speed, seek time, transfer rate, scsi card, and number of other devices in the chain. The speed of the mirroring will be equal to the slowest variable.

I would recommend that you mirror to ???LIKE??? devices. Internal disks to internal disks, external disks to external disks???..


Live free or die
harry