unmount file systems prior to package start

[Greg Geyer]

Unfortunately we have 3 file systems on the failover server with the same names as filesystems in the package. The business/developers have declined to rename these, so any automatic failover would fail upon trying to mount over these already mounted filesystems.

The customer functions in the control file come after the LV mounts so that would not provide me an opportunity to fuser/unmount those filesystems.

I'm looking at several options to try and was wondering if anyone had already solved this issue:

1) In the package config file redirecting the RUN_SCRIPT to point to another script, which will do the fuser/unmounting and then call the package control file. I'm concerned that the cmapplyconf won't like this "control file wrapper" setup.

2) make the 3 filesystems on the failover server their own package and have that package halt (no failover) prior to the production package starting there. I like this idea but I see no mechanism in MC/SG to direct one package to stop before another will start.

Thanks - Greg

[Jeff_Traigle]

As far as option 2 that you mention, that's handled within the customer defined halt/run sections of the package control script. That's actually covered in the Serviceguard class. Just run the cmrunpkg/cmhaltpkg commands as you need them for whatever other package you need to manipulate from the this package.

[Tiziano Contorno _]

I'm thinking about mounting the non clustered FS on different mount point and using links.

/app link to /app_noncl

FS_noncl mounted on /app_noncl

This way you need to remove a link, no unmounting.

Not a solution to your question but maybe an alternative to consider.

Regards.

[Greg Geyer]

Jeff -> the customer run functions, as I stated originally, occur AFTER the LV portion of the control file. So the package would fail long before it gets to the customer run scripts. Or are you suggesting something else?

Vanadio -> you're right it doesn't actually solve the problem but I like the suggestion in that it is one less step (rm the link instead of fuser/umount) and probably would work more often.

[Greg Geyer]

Well there's an unsupported way, but it works. In the control file there's the function "freeup_busy_mountpoint_and_mount_fs".

If you're FS_MOUNT_RETRY_COUNT is >0 then this function fuser's anyone on your mount point. I added a "umount ${mount_pt} immediately after the fuser command and it worked (on a test cluster). My addition is between the double-comments:

function freeup_busy_mountpoint_and_mount_fs
{

typeset vol_to_mount
typeset mount_pt
typeset fs_mount_opt

vol_to_mount=$1
mount_pt=$2
shift 2
fs_mount_opt=$*

print "\tWARNING: Running fuser on ${mount_pt} to remove anyone using the busy
mount point directly."
UM_COUNT=0
RET=1

# The control script exits, if the mount failed after
# retrying FS_MOUNT_RETRY_COUNT times.

while (( $UM_COUNT < $FS_MOUNT_RETRY_COUNT && $RET != 0 ))
do
(( UM_COUNT = $UM_COUNT + 1 ))
fuser -ku ${mount_pt}
######
######
umount ${mount_pt}
######
######
if (($UM_COUNT == $FS_MOUNT_RETRY_COUNT))
then
mount ${fs_mount_opt} ${vol_to_mount} ${mount_pt}
(( RET = $? ))
if(( $RET != 0 ))
then
print "\tERROR: Function freeup_busy_mountpoint_and_mount_fs"
print "\tERROR: Failed to mount ${vol_to_mount} to ${mount_pt}"
break
fi
else
mount ${fs_mount_opt} ${vol_to_mount} ${mount_pt}
(( RET = $? ))
sleep 1
fi
done
return $RET
}

[A. Clay Stephenson]

This is state of the art stupid. It sounds as though you have development and production going on - on the same box -- at least upon package failover. If these boxes fall under SOX then you are going to get dinged bigtime. Even if you umount the "bad" filesystems (which could be very iffy) and mount the "good" filesystems, what prevents a development user from connecting to the production package? The mountpoints are the same. How would he know?

If you guys have application hard-coded pathnames then this is less than robust. Ideally, the pathnames would be based upon some environment variable so that naming the filesystems is no big deal. Production and development would have different environment variable settings so that collision would now not be possible.

Your problems appear to extend far beyond those of simple mounting issues.

[Greg Geyer]

Clay,

I agree. I've been shot down by the powers that be about changing file system names.

Excellent point about the SOX issues, as we are under that gun. I'll bring that issue up to the business.

Thanks - G.

[Geoff Wild · Accepted solution]

Here's how to get around it - create a new function:

At the bottom of the cntl file, add it before the activate_volume_group:

customer_defined_umount

activate_volume_group

Under "# START OF CUSTOMER DEFINED FUNCTIONS" put your function something like:

function customer_defined_umount
{
print "umount filesystems because some developers are....well...you get the idea..."
fuser -cku /mount1
sleep 5
fuser -cku /mount2
sleep 5
fuser -cku /mount3
sleep 5
umount /mount1
umount /mount2
umount /mount3
test_return 51
}



Rgds...Geoff

[Jeff_Traigle]

If you're trying to run the halt on the second package in the start for the other, yes, that will be a problem.

Say you have package A that is the one you need to move. Say Package B is the one you have the mounts you need to umount before Package A can start.

In Package A halt section, run cmhaltpkg on Package B. Everything is umounted before Package A ever tries to start on the other node.

Granted, not ideal way to be handling this. Always best to have unique mount points, but it can be handled within the control script framework with some messy checks and balances coded into it.

[Greg Geyer]

Geoff -> I like it. I'll try it on my test cluster.

Jeff -> Not bad too. The only problem I see with it is if a hard crash occurs and cmhaltpkg can't do its work. Still your idea would take care of most issues.

Clay -> Good news is that the main package actually does shut down development (using the SAP toolkit). But that does not deny the fact that the developers could have ftp jobs or something else run even when dev is down. So I just went back with the SOX argument. Might as well use it to my advantage.

Thanks to the excellent responses by all.

G.

[A. Clay Stephenson]

But all of this misses the fundamental point; it is now possible for a development/test user to connect to production and possibly not even know it. This even assumes that all the umounts and mount precautions work perfectly. In fact, those procedures actually serve to make the situation more dangerous -- simply because they work. Be glad I'm not your internal auditor; be very glad I'm not your external auditor.

I restate my position more diplomatically. It's still state of the art stupid.

[Jeff_Traigle]

One problem with Geoff's suggestion is that it's not within the SG norm of how things are handled. All well and good if you're the only one dealing with the system until it goes away and can remember the changes you made. But what if someone else comes along and needs to change the package? (Say some upgrade to SG comes along that recommends this to take advantage of some changes HP makes to the control script.) They could come along and run "cmmakepkg -s" and copy the customer defined sections from the current control script as HP's documentation would most likely point out, but never give a thought that someone hacked HP's control script to do other things outside of these customer defined sections. Then everyone's spending time and effort trying to figure out why the packages don't work any longer. (Yeah, you can document your changes like you should, but we all know documentation for these standard routine tasks don't always get read... probably rarely actually... people even lose the knowledge that such documentation exists in an environment.)

Sure, it'll work, but it's not a very maintainable approach within standard guidelines.

[Greg Geyer]

I agree with you Jeff and Clay. I'm only trying these options because of the corner I am in. The people who deny me changing filesystem names will be the same people who complain either about manual failover, or an overwritten file.

SOX will be my sword in this fight since I've been reminded of this tool. I don't know why I would be afraid of Clay being my internal auditor, I don't want it this way.

[Greg Geyer]

Thanks for the replies, as we're down to the gnashing of teeth part of the discussion.