Serviceguard mount/umount problem

[Srinikalyan]

Hi all,
we have around 72 filesystems(around 69 for Oracle mount points) to mount and umount using Serviceguard packages in the production system. Serviceguard package is just to start/stop Oracle service. It's taking lot of time around 45 mins to start/stop packages. If I start/stop Oracle by Oracle commands it's taking only 2 to 3 mins time. I suspect there is a problem in mounting/unmounting the 72 filesystems. In the package cntl comments, I am seeing by changing the CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS, system will see performance. We have lots of resources free at the time of planned maintenance. Can anyone help me which value for CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS will suit this environment?
Note: Currently no logs available.(Sorry for that)
HP-UX 11iv2, Serviceguard A.11.17,
ia64, 32 Dual core CPU's
Thanks,
Srini

[Steven E. Protter]

Shalom,

Can you pull the volume group activation code and the file system mount mount code from the packages and post an example?

Perhaps someone can spot the problem based on experience.

SEP

[Srinikalyan]

As requested, at first I will give the umount scripts( Since stopping the packages takes lof of time)
CONCURRENT_VGCHANGE_OPERATIONS=1
CONCURRENT_FSCK_OPERATIONS=1
FS_UMOUNT_COUNT=1
FS_MOUNT_RETRY_COUNT=0
CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS=1
# Unmount each logical volume.

function umount_fs
{
integer UM_CNT=${FS_UMOUNT_COUNT:-1}
integer ret
integer j

set -A LogicalVolumes ${LV[@]}

if [[ $UM_CNT < 1 ]]
then
UM_CNT=1
fi

integer L=${#LogicalVolumes[*]}

# Perform parallel file system umounts for better performance.
# Limit the number of parallel umounts to CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS

typeset mounttab=$(mount)

typeset pids_list
typeset volume_list

while (( L > 0 ))
do
j=0
while (( j < CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS && L > 0 ))
do
(( L = L - 1 ))
I=${LogicalVolumes[$L]}
echo $mounttab | grep -e $I" " > /dev/null 2>&1
if (( $? == 0 ))
then
print "$(date '+%b %e %X') - Node \"$(hostname)\": Unmounting filesystem on $I"
(
umount ${FS_UMOUNT_OPT[$L]} $I; ret=$?
if (( ret != 0 ))
then
print "\tWARNING: Running fuser to remove anyone using the file system directly."
fi

UM_COUNT=$UM_CNT
while (( ret != 0 && UM_COUNT > 0 ))
do
fuser -ku $I
umount ${FS_UMOUNT_OPT[$L]} $I; ret=$?
if (( ret != 0 ))
then
(( UM_COUNT = $UM_COUNT - 1 ))
if (( $UM_COUNT > 0 ))
then
print "\t$(date '+%b %e %X') - Unmount $I failed, trying again."
sleep 1
fi
fi
done
return $ret
) &

# save the process id and name of logical volume to be used later
# while checking the exit status
pids_list[$j]="$!"
volume_list[$j]=$I
(( j = j + 1 ))
fi
done

# wait for background umount processes to finish
# I is used by "test_return 13"
while (( j > 0 ))
do
pid=${pids_list[$j-1]}
I=${volume_list[$j-1]}
wait $pid
if (( $? != 0 ))
then
let 0
test_return 13
fi
(( j = j - 1 ))
done
done
}


function deactivate_volume_group
{
# Perform multiple volume group deactivations at same time.
# Limit the number of concurrent deactivations to CONCURRENT_VGCHANGE_OPERATIONS

integer index=0
integer j
integer ret
integer num_retries=0
set -A VGS ${VG[@]}
integer num_vgs=${#VGS[*]}

typeset pids_list
typeset volume_list

while (( index < num_vgs ))
do
j=0
while (( j < CONCURRENT_VGCHANGE_OPERATIONS && index < num_vgs ))
do
I=${VGS[$index]}
print "$(date '+%b %e %X') - Node \"$(hostname)\": Deactivating volume group $I"
(
vgchange -a n $I; ret=$?
while (( ret != 0 && num_retries < DEACTIVATION_RETRY_COUNT ))
do
print "\t$(date '+%b %e %X') - vgchange -a n $I failed, trying again."
if [[ $KILL_PROCESSES_ACCESSING_RAW_DEVICES = "YES" ]]
then
print "\tWARNING: Running fuser to remove anyone using the raw device directly."
find /dev/${I##*/} -type c \! -name group | xargs fuser -ku
fi
sleep 1
vgchange -a n $I; ret=$?
(( num_retries = num_retries + 1 ))
done
return $ret
) &
# save the process id and name of VG, used while checking the exit status
pids_list[$j]="$!"
volume_list[$j]=$I
(( j = j + 1 ))
(( index = index + 1 ))
done

# wait for background vg deactivations to finish
# I is used by "test_return 14"
while (( j > 0 ))
do
pid=${pids_list[$j-1]}
I=${volume_list[$j-1]}
wait $pid
if (( $? != 0 ))
then
let 0
test_return 14
fi
(( j = j - 1 ))
done
done
}

[likid0]

Hy,

As you say you can try pumping up the CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS values.


I have a system with 61FS and I use:
CONCURRENT_VGCHANGE_OPERATIONS=X <-- number of vgs you have in the package
CONCURRENT_FSCK_OPERATIONS=61
CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS=61

[Srinikalyan]

Thanks. Closing the thread.