How to implement SCSI reserve/release

[Olivier Masse]

I'm trying to develop a cluster lock mechanism similar to the one used by MC/ServiceGuard, but by using shared raw devices directly. This is a personal project as I would like to cluster my development systems and MC/S is way overkill for that.

From my understanding, this is implemented by the master node doing a SCSI reserve on a shared lock disk. If there's any network outage, when the cluster reforms, a scsi reset is sent to clear any outstanding lock, then the first node to acquire the lock can restart the services.

I found that the scsi_pt driver seems to let me do this somewhat. I hacked the /usr/contrib/src/scsi_io.c example code to send SCSI reservations but they seem to go in the blue, it doesn't do anything as I can acquire would-be locks from both nodes on the same disk. I'm not a hotshot C programmer so something is obviously not right.

I wondered if there is some example C code somewhere that would show me how to implement this.

Thank you

[Decio Miname]

SCSI reservation is one of the ways to implement disk-based concurrence control (heartbeat). Since you are implementing this feature by yourself, I believe it is a great time to look at the Linux Advanced Server (currently named AS version) cluster features. It describes very well this and some other forms of concurrence control, even being at high level only. Particularly, I like the poll/write counter based on raw access that AS uses - it looks very hardware-independent.
I've seen some problems of undesired hangups due to the SCSI reservation technique on IBM AIX and EMC Symmetrix. Bottom line is, each side is "intelligent" on dealing with the reservations, and in a few occasions one side thinks itself as smarter than the other, resulting in a hang that could not be recovered without hard reset on the bus.

[Olivier Masse]

I'll take a look at Linux AS, if it is well documented it will give me an idea on how to do that. I'll dig through Red Hat's site to see if they have technical information.

I've been thinking about this during the weekend and was considering concurrent writes on unique records on a shared raw disk instead. For example, node A knows that it can write only to record #1 and node B to record #2. All the records can be read by the nodes to see the status of the other one. I don't know if it will work, but it's easy to implement to try it out.

Thank you Decio.

[Decio Miname]

Yes, that's the idea. It should work. Just check if the SCSI buses you'll be suitable to use don't do concurrence control for themselves - which could be undesired for your implementation. On HP-UX I believe (but I am not sure) that should NOT be a problem, since this feature is available only if you had MC/SG (-c flag on vgchange).
Good luck!
D.