FC10's, Mirroring, and Alternate Paths

[Fred G. Claypool, Jr.]

Hello everyone,

Here's some background...
-------------------------

We currently have three FC10's (fibre channel disk systems) attached to our K570. This K570 serves as our production (Informix) database server. Each FC10 has two LCCs; we follow the "No-Hub Configuration Topology"; and each LCC is connected via fibre cable directly to our K570. Two of the FC10's are populated with 36 GB drives and the older one has 18 GB drives. Each drive is configured via LVM as a physical volume using its Primary Path (let's just say the top LCC) as its Primary Link and its Alternate Path (let's just say the bottom LCC) as its Alternate Link. FC10-1 contains primary extents, FC10-2 contains mirror extents, and FC10-3 (the older FC10) contains its primary extents on disks 0-4 and its mirror extents on disks 5-9.

In the past few months, we've had occasional drive failures. In our past failure, we could access one of the 36 GB drives in question via its Alternate Path (using diskinfo), but not via its Primary Path (using diskinfo). During this time, our Informix DB was essentially stopped dead in its tracks. Our HP engineer recommended as a short-term "try this" to pull this drive, wait for a minute, and then reinsert it. Miraculously, our problem was solved.

Our original intent when these disk systems were configured was to have as much redundancy built-in as possible given this single DB host. Our data is mirrored (= +1 bonus point) and we even have an alternate link to each disk (= +1 bonus point).

Our ASE is currently performing a whole-landscape patch analysis to make sure we're up-to-date, especially with our LVM software.

Now the questions!
------------------

1) There has been recent debate in our HP support community regarding our use of Alternate Links. Our former ASE says that Primary Links & Alternate Links must always point to the same physical disk device, but our new ASE says that they should always point to different physical disk devices. Which is it? (Can BOTH be right given the right landscape?)

2) Our former ASE says that none of her other customers utilize BOTH Mirroring AND Alternate Links. Can these two beasts coexist, or are we trying to be TOO redundant?

3) Is it acceptable or totally crazy to place a logical volume's primary extents on 1/2 of an FC10 with that same logical volume's mirror extents on the other 1/2 of that same FC10?

4) Consider a non-Alternate Link environment for a moment. If mirroring is employed, will LVM automatically utilize the mirror extents for reading if the primary extents are busy being written to? Or does the whole concept of mirroring simply mean that your data is safe if something happens to your primary extents?

Thank you all for your advice. We also really appreciate our past & present ASE's, but I want to invite a greater audience to share your experiences too.

~Fred Claypool, Jr.

[Carsten Krege]

1) LVM identifies disks by its physical volume id (PVID) and will only recognize a disk as alternate link if it is on a different hardware path, but has the same PVID. Therefore alternate links should always point to the identical disk device (exception: copies of LVM disks made by dd for example, EMC BCVs, XP business copies. All these are copies of disk on a level below LVM).

2) It is very common and recommended to use both mirroring and alternate links.

3) This form of redundancy is not as good as if it were two different boxes you would have primary and secondary copies on. However, it is again very common that customers do this and it is still a high level of redundancy. If your complete FC10 fails you will have a problem though.
You could configure your mirrors in a way that the copies are always on a different FC10 (take a piece of paper and draw it first). Would be quite an effort to change the config though.

4) No, if the primary copy is unavailable, the other half of the mirror should be accessed online and automatically. What you have seen is a misbehaviour and must be fixed.

Carsten

[Fred G. Claypool, Jr.]

Carsten,

Thanks for your reply! I really appreciate your advice.

Concerning your fourth response, should LVM respond (meaning: automatically & quietly access the mirror data with no interruption to our applications) the same in an Alternate Link environment as your response indicates it should in a non-Alternate Link environment?

Additionally, concerning the "misbehaviour" of our current situation, where would you recommend we look for advice? (HP-UX patches, LVM configuration, etc.?) Have you ever heard of such an LVM environment (mirrored, alt link) causing such a work stoppage that is "fixed" simply by removing & replacing the same disk device?

Thanks again,
~Fred

[Fred G. Claypool, Jr.]

Drat! I thought of another question right after I sent that reply! :-)

The only reason I was considering making ALL THREE of our FC10's follow the "1/2 primary extents -- 1/2 mirror extents" model was because that's how our older FC10 is configured right now. I could easily change this such that each FC10's disks 5-9 were the mirror of the previous FC10's disks 0-4. However, that new model will only work if I exchange some of the 36GB drives with some of the 18GB drives -- which means that two of my three FC10's would contain a mixture of drive sizes. Is this Ok? Anything at all that I should worry about?

Thanks,
~Fred

[Carsten Krege · Accepted solution]

Mirrored config
If a disk fails, the data should be accessed transparently on the mirror copy. The user wouldn't notice anything, in spite of messages that should be logged that indicate a failure of the disk.

Alternate links
This is a feature that is completely independent from MirrorDisk/UX. It provides redundancy if anything on the h/w path to a disk fails, e.g. host adapter, cable etc. If one of these components fails, LVM will be able to access the disk via the alternate link.
If a disk fails in a mirrored/alternate link environment, again, the data is transparently accessed on the mirror copy. The features are really independent.

From the few data I have it is not clear, what particularly happened on your system. It could be nearly anything (Hardware, LVM, FC, SCSI driver, WSIO). I would recommend to let the HP RC perform a careful patch analysis that includes patches for all of these subsystems and I would recommend to test whether mirroring works (i.e. when you have a downtime, put load on a disk (read and write!) and then pull one of the mirror copies. I/O should continue now) after the patches were applied. I also strongly recommend to check the firmware levels of the disk drives.

I cannot say whether whether you can put 36GB mechs in your 18GB mechs cabinet (and vice versa). I'm pretty sure that it should work, but this must be evaluated by the HP h/w folks. Sometimes it is necessary to have the right firmware on disks and controller hardware and of course the drive height must be considered if they are different.

Carsten

[Fred G. Claypool, Jr.]

Carsten,

Thanks again for your advice. I think we're now in a better position to understand our options, and hopefully that patch analysis will bring us some comfort.

If anyone else out there in the ITRC Forums has anything to add, I'll be much appreciative of your post.

Thanks,
~Fred