LVM Consistency after failover

[Mark van Hassel]

I have a discussion with some of my colleagues and disk array vendors about this subject.
When the primary host is performing writes when it crashes, Is the data, when activated on the standby system consistent and not corrupt ? I know there is a Mirror Write Cache, but that is in memory and only written to disk occasionally, so the standby system is not up to date. By default are writes synchronous, i.e. when you receive write completed is the data only in cache or on disk ?

I think there is no problem with databases after failover due ti their recovery mechanisms, but there could be data loss and/or corruption on file systems after a fail over occurred.
Is this correct ?

Thanks,

Mark

[Stefan Farrelly]

My understanding is this;

External disk arrays now guarantee that writes will be committed once sent to them by a host. They pass back to the host immediately that the write has been done (for performance) even though its in cache on the array and may not yet be committed to physical disk - but disk arrays nowadays have battery backup or dual power etc. so they will always guarantee that the write is done even (albeit slightly delayed) in the event of a powerfailure to the array.

So, your data is always guaranteed to be written to disk. If one server dies and another picks up the same disk shortly after then yes - HP-UX will have to fsck the filesystem as they will be in a slightly inconsistent state (as you would get normally if your single server crashed). But with JFS and logging this is fast and reliable so no problems there.

But - for databases, in my experience, you can still get corruptions if this happens. Weve had this happen to use twice this year. Despite using an EMC array when a server crashes the filesystems come up fine (fsck works aok) but the database detects a corruption in its data or indexes. In our case I think this is caused by us not running our databases in archive log mode - which would allow the db to simply rollback successfully after a crash. Management here were for some reason of the opinion that an EMC disk array would guarante no corruptions - but this is not the case. In my opinion its always safer to run your db's in archive log mode for additional protection against data corruption. Other servers of ours running archive log mode have never been corrupted after a crash. So - use the recovery mechanisms within your databases as well as using an external disk array.

[John Palmer]

Hi,

There is no guarantee that all data has been written to disk, because HP-UX buffers so much disk data in the buffer cache for performance.

The application can however insist on data being written to disk and this is how database engines such as Oracle write data to their redo logs so that they can perform recovery in the event of a crash.

Mirror Write Cache only protects the LVM consistency of a logical volume, it doesn't protect data.

You do have various mount options etc available that can influence how data is cached and written to disk but it's always a trade-off between performance and security.

From your mention of primary and standby, it sounds as though you have a Serviceguard environment. A successful implementation requires an application that is capable of recovering from system crashes, it's no use if your app fails over to the standby but then needs manual intervention to fix the data.

Regards,
John

[Mark van Hassel]

Thanks for the replies. Yes this concerns a service guarded environment.
I've been searching through the forums for performance issues and I noticed that it is best to turn of MWC, but recovery on. Out of curiosity: how does LVM know which extent is current and which mirror shoul be updated ? From a timestamp ?

There's also some issue with the mount options delaylog/nodatainlog to enhance performance on archive/redo log files and datafiles. Should I use this in a MCSG cluster ? Or is it safer to use the log/datainlog options to ensure no corruption. Or should I not worry about en let the Oracle database repair itself ?

Thanks,

mark

[John Palmer]

If you turn off MWC, LVM will completely resync the volume in the event of a crash. As to which mirror it will select to sync from, it doesn't matter because they'll both be in an uncertain state.

Oracle is very good at recovering after a crash, it's one reason why you pay a lot of money for it ;-) You don't have to use special mount options for Oracle but if you have online JFS you can use them to improve performance. Alternatively, you can use raw logical volumes and let Oracle sort things out.

Regards,
John