system hangs at Mount file systems .......

[ran_li]

Hello all, need a clue to troubleshoot system hangs at start up time. I m able to enter single user mode, ran the fsck then either go to runlevel 2 or reboot the system, it will hangs at Mount file systems .... stage, please give a direction. Thanks.

# init 2

INIT: New run level: 2
/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" has been successfully changed.
vxfs fsck: sanity check: root file system OK (mounted read/write)
Checking hfs file systems
/sbin/fsclean: /dev/vg00/lvol1 (mounted) ok
HFS file systems are OK, not running fsck
Checking vxfs file systems
/dev/vg00/lvol8 :
vxfs fsck: sanity check: /dev/vg00/lvol8 OK
/dev/vg00/lvol3 :
vxfs fsck: sanity check: root file system OK (mounted read/write)
/dev/vg00/lvol4 :
vxfs fsck: sanity check: /dev/vg00/lvol4 OK
/dev/vg00/lvol5 :
vxfs fsck: sanity check: /dev/vg00/lvol5 OK
/dev/vg00/lvol6 :
vxfs fsck: sanity check: /dev/vg00/lvol6 OK
/dev/vg00/lvol7 :
vxfs fsck: sanity check: /dev/vg00/lvol7 OK

Cleaning /etc/ptmp...
(c)Copyright 1983-2000 Hewlett-Packard Co., All Rights Reserved.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-1992 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1986 Digital Equipment Corp.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2000 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.
(c)Copyright 1991-2000 Isogon Corporation, All Rights Reserved.


RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.

Hewlett-Packard Company
3000 Hanover Street
Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).

/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)


HP-UX Start-up in progress
__________________________

Configure system crash dumps ..............................................
.. OK
VxVM device node check ....................................................
.. OK
VxVM general startup ......................................................
.. OK
VxVM reconfiguration recovery .............................................
.. OK
Mount file systems ........................................................
..

[Mel Burslan · Accepted solution]

Check if you have any NFS mounts made from the fstab and if yes check if the nfs server(s) is/are alive for these mount points. It does not seem like you have any corrupt local volumes so, this leaves some external (NFS/CDFS/etc.) filesystems to blame.

[Paul Sperry]

I think Mel is correct. It's probably an old/missing nfs mount. Check your /etc/fstab
and comment them out (you can allways mount them later)

[ran_li]

Hi guys, thanks for the thoughts but it does not have any nfs mount

# cat /etc/fstab
# System /etc/fstab file. Static information about the file systems
# See fstab(4) and sam(1M) for further details on configuring devices.
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /home1 vxfs delaylog 0 2
/dev/vg00/lvol5 /opt vxfs delaylog 0 2
/dev/vg00/lvol6 /tmp vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2

nor does it a NIS client.

I can mount all of the volumes manually in single user mode, how come it will stack during the boot.

[Tom Smith_9]

Have you made any changes to /etc/fstab recently? Try to recreate the fstab manually and see if that helps.

-Tom

[ran_li]

No, the /etc/fstab was not touched by anybody, plus it's identical to other boxes which can be booted up. any other suggestions?

[Mel Burslan]

a suggestion:

can you try booting to single user and mount the filesystems one-by-one manually to see what the heck is going on. With the barebones filesystem structure, I do not see any reason why it should hang like that.

[ran_li]

when I entered single user mode

# mount
/ on /dev/vg00/lvol3 log on Tue Sep 6 16:24:05 2005
/stand on /dev/vg00/lvol1 defaults on Tue Sep 6 16:24:06 2005

after mounting everything manually

# mount
/ on /dev/vg00/lvol3 log on Tue Sep 6 16:24:05 2005
/stand on /dev/vg00/lvol1 defaults on Tue Sep 6 16:24:06 2005
/home1 on /dev/vg00/lvol4 log,nodatainlog on Tue Sep 6 16:25:25 2005
/opt on /dev/vg00/lvol5 log,nodatainlog on Tue Sep 6 16:25:53 2005
/tmp on /dev/vg00/lvol6 log,nodatainlog on Tue Sep 6 16:26:09 2005
/usr on /dev/vg00/lvol7 log,nodatainlog on Tue Sep 6 16:26:27 2005
/var on /dev/vg00/lvol8 log,nodatainlog on Tue Sep 6 16:27:02 2005

no error was reported.

after init 2 or reboot, it will hang at "Mount file systems ... " like before.

I think it appears stopping at "Mount file system ... " but have other reasons, not only fstab, sorry I don't have a clue.

[Devender Khatana]

Hi,

/sbin/init.d/localmount script is used for mounting file systems while OS is booting. What happens if you try to mount file systems using this script inspite of mounting manually.

Try in single user mode and observer the output.

#sh -x /sbin/init.d/localmount


HTH,
Devender

[Mel Burslan]

Ran,

Under the circumstances, you have a little investigation to do. edit the file:

/sbin/init.d/localmount

and just before the first uncommented line, insert another line like this

set -x

save it and reboot your machine. when it attempts to mount all filesystems, server will execute this command and will give you an output of every command it executes or tries to execute in this little script. It will most probably mess up your screen but you can live with this just this once. Watch the screen (capture it if possible using a terminal emulator and its logging capabilities) very closely as the commands execute and see where the execution stops and waits. You may be running into a misconfigured quota situation. If you can get any reasonable screen capture, feel free to post them here for further help.

Last but not the least, you have not assigned a single point to any of the 18 responses you get from people on these forums. They took time away from their days to try helping you resolve your problem. You can at least show siome courtesy to assign points to those who helped you resolve your problems. The questions with unassigned points can be found here:

http://forums1.itrc.hp.com/service/forums/pageList.do?userId=CA1294147&listType=unassigned&forumId=1

[ran_li]

Hi Mel, Devender and all,

I always appreciate you guys' help, just don't know this rule here, I will append the points for all my post, the link you give it to me does not work in my browser, I will find all my posts. Thanks for the reminder.

the scripts stopps at quotacheck

# /sbin/sh -x /sbin/init.d/localmount start
+ PATH=/sbin:/usr/sbin:/usr/bin
+ export PATH
+ rval=0
+ /sbin/mountall -l -m
+ set_return
+ [ -f /quotas -a -x /usr/sbin/quotaon ]
+ [ -x /usr/sbin/quotacheck ]
+ echo checking quotas
checking quotas
+ awk
$1 !~ /^#/ { if ($3)
if ($3 != "swap" && $3 != "noswap" && $3 != "ignore")
types[$3] += 1
}
END { for (i in types)
printf ("%s ", i);
printf ("\n");
} /etc/fstab
+ quotacheck -F hfs -a -P

then hangs here.

Regards!

[Mel Burslan]

Ran,

If your system is version 11i, I do not see a -P option for quotacheck command on the manpages for the command.

Unless you are using and enforcing quotas, try commenting the quotacheck command lines in the localmount script and try rebooting hopefully one final time.

When youare all done, if you modified this script localmount as I suggested above, you may want to remove those set -x statements before completing your work.

[ran_li]

Thanks Mel, I can go to runlevel 2 and 3 now, I commented out following lines,

# hfs) quotacheck -F $i -a -P # preen quota stats
# set_return
# ;;

Just curious, how come I have other 11i boxes which with exactly the same lines in the /sbin/init.d/localmount do not have this kind of problem at all, besides, this host was working properly before, this just happened all of sudden.

I also tried remove the -P option but it does not work,

hfs) quotacheck -F $i -a # preen quota stats
set_return
;;

everything is the same like before (hangs at Mount file systems ...), unless I comment out all above 3 lines.

Regards!

[Mel Burslan]

it is a wild idea but either you are missing a patch to fix quotacheck command on this server or someone migh have brought the quotacheck command from another server, which may be incompatible. Check the filesize of the command between this server and a server you know it is working (make sure this server is running the quotacheck command in its localmount script first)