How to reap fc60mon?

[Ralph Grothe]

Hi,

I've got the following problem.
In order to increase the filesystem mounted as /opt I have to unmount it (unfortunately on this box there is no OnlineJFS fsadm extension available due to licensing cost).
After having safely shut down all process that accessed the /opt filesystem I still cannot get rid of this hiddeous fc60mon process.
Even when I forcebly try to kick it out it somehow automagically restarts:

# ps -fp "$(fuser -cu /opt 2>/dev/null)"
ps: wrong PID number
UID PID PPID C STIME TTY TIME COMMAND
root 22814 22813 0 10:01:26 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/fc60mon

# fuser -k /opt;umount /opt
/opt:

umount: cannot unmount /opt : Device busy

ps: wrong PID number
UID PID PPID C STIME TTY TIME COMMAND
root 22814 1 0 10:01:26 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/fc60mon

# kill -9 22814
# kill -0 22814
kill: 22814: The specified process does not exist.
# umount /opt
umount: cannot unmount /opt : Device busy


After a quick consultation of fc60mon's manpage I also shut down EMS monitoring by the monconfig utility.
Still fcmon60 insolently reappears.

Any remedy?

[S.K. Chan]

It has to be the /etc/inittab entries respawning EMS, that's why it keep re-appearing. Check /etc/inittab, I think the entries would have "ems" as it's first field follow by colon (:) and the run levels. Make a copy of the inittab file then delete those entries and re-read inittab ..
# init q
Hopefully that'll work for you ..

[Ceesjan van Hattum]

A simple answer from a simple person like me:
$> init 1
$> cd /; umount /opt
$> lvextend -L NNN /dev/vg00/lvolX
$> mount /opt
$> init 2

Assuming EMS and others are in runlevel 2 or higher..

Regards,
Ceesjan

[Yogeeraj_1]

Hello,

this is how i would go about increasing the size of /opt

Follow these steps to increase the size limit of /opt:

Determine if any space is available:

/sbin/vgdisplay

You should see output like this:
- Volume groups -
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 8
Open LV 8
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 2000
VGDA 2
PE Size (Mbytes) 4
Total PE 249
Alloc PE 170
Free PE 79
Total PVG 0
The "Free PE" indicates the number of 4MB extents available, in this case 79 (equivalent to 316 MB).

Change to single user state:

/sbin/shutdown

This will allow /opt to be unmounted.

View mounted volumes:

/sbin/mount

You will see a display similar to the following:

/ on /dev/vg00/lvol1 defaults on Sat Mar 8 23:19:19 1997
/opt on /dev/vg00/lvol6 defaults on Sat Mar 8 23:19:28 1997


Determine which logical volume maps to /opt. In this example, it is /dev/vg00/lvol6

Unmount /opt:

/sbin/umount /opt

This is required for the next step, since extendfs can only work on unmounted volumes.

Extend the size of the logical volume:

/sbin/lvextend -L new_size_in_MB /dev/vg00/lvol6

For example, to make this volume 332 MB:

/sbin/lvextend -L 332 /dev/vg00/lvol6

To extend the file system size to the logical volume size:

/sbin/extendfs /dev/vg00/rlvol6

Mount /opt:

/sbin/mount /opt

Go back to the regular init state: init 3 or init 4, or reboot.


hope this helps

best regards
yogeeraj

[Ralph Grothe]

Chan,

your suggestion was well worth a try, though it didn't really help :-(

Instead of commenting the whole ems entry from /etc/inittab I simply substituted the 3rd field with an "off" value for the "respawn" value, and issued an "init q".

At least it helped that much that the fc60mon daemon isn't respawned again, and an "fuser -cu /opt" doesn't show any processes anymore.

But when I try an "umount /opt" I still get the dreaded "device busy".
Looks as if some open file, file lock, or such is still lingering around which even gets unnoticed by fuser.
I haven't the GNU lsof installed on this box, though.
Maybe I should do so to possibly get a more detailed view on /opt?

[Ralph Grothe]

Ceesjan, Yogeraj,

thanks for your efforts, but I do know the basic procedures how to extend filesystems from single or maintenance mode.
The issue is that currently I cannot bring down the machine in such a run level.

But now I know from a sw call request at the HP support that I definetly will have to down the machine, as apart from /, /var, and /usr (sometimes) it is also required for /opt to do so in order to be able to umount the filesystem :-(

[Victor BERRIDGE]

Hi,
Are you sure you dont have another mount point on /opt? that you havent noticed? like at the bottom of bdf, some /opt/patrol ...
It happened to me once, I saw all the oracle mount points etc..
and missed that one which was added durinig a vacation...

Just thoughts
All the best
Victor

[Yogeeraj_1]

Hello,

maybe these hints might help.

It seems like the process is forked by one of the scripts found in /usr/sbin/stm/xxx directory.
-----------------------------------------------
# ps -ef|grep fc60mon
root 2651 1 0 Jun 9 ? 0:04 /usr/sbin/stm/uut/bin/tools/moni
tor/fc60mon
root 580 1858 0 17:14:00 pts/tc 0:00 grep fc60mon
# grep -i "fc60mon" *
# grep -i "stm" *
diagnostic: if [ -x /usr/sbin/stm/uut/bin/progs/PVSU ] # Stable stora
ge solution
diagnostic: /usr/sbin/stm/uut/bin/progs/PVSU 2> /var/stm/logs/pvsui
nitlog
diagnostic: if [ -a /usr/sbin/stm/uut/bin/sys/diagmond ]
diagnostic: if [ -x /usr/sbin/stm/uut/bin/sys/diagmond ] &&
diagnostic:/usr/sbin/stm/uut/bin/sys/diagmond
diagnostic: print -u2 "/usr/sbin/stm/uut/bin/sys/diagmond failed
during"
diagnostic: if [ -a /usr/sbin/stm/uut/bin/sys/diagmond ]
diagnostic: # Assume stm is shutdown
already
omni: CRSTMP=${OMNIHOME}/tmp
omni: CRSTMP=${OMNIHOME}/tmp
omni: cd ${CRSTMP} 1> /dev/null 2>&1
supprtinfo:export DIAG_SYS=/usr/sbin/stm/uut/bin/sys/diagmond
#
-----------------------------------------------
I would run all these scripts found in the /sbin/init.d with argument stop!
e.g. omni stop
diagnostic stop
etc...

Hope this helps..

regards
yogeeraj