topic Re: VxVM startup errors in rc.log in Operating System - HP-UX

VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 15:50:26 GMT

I tried to go into single user mode on one of our servers to do some maintenance on file systems and when I tried to mount my current volumes was receiving messages:

vxfs mount; Cannot open /dev/vx/dsk/rootdg/tmpvol ; No such device or address

I received this on tmp, var, opt, usr, etc..... and I was able to cd into /dev/vx/dsk/rootdg and all the special files for the volumes are there:

[/root] # cd /dev/vx/dsk/rootdg
cnpuref1[/dev/vx/dsk/rootdg] # ll
total 4
drwxr-xr-x 2 root root 1024 Feb 12 02:48 .
drwxr-xr-x 3 root root 1024 Feb 12 02:48 ..
brw------- 1 root root 2 0x000007 Feb 12 02:48 homevol
brw------- 1 root root 2 0x000006 Feb 12 02:48 optvol
brw------- 1 root root 2 0x000000 Feb 12 02:48 rootvol
brw------- 1 root root 2 0x000005 Feb 12 02:48 standvol
brw------- 1 root root 2 0x000001 Feb 12 02:48 swapvol
brw------- 1 root root 2 0x000008 Feb 12 02:48 tmpvol
brw------- 1 root root 2 0x000009 Feb 12 02:48 usrvol
brw------- 1 root root 2 0x00000a Feb 12 02:48 varvol

I rebooted the server back up and noticed following errors in rc.log for VxVM:

VxVM device node check
Output from "/sbin/rc1.d/S091vxvm-nodes-check start":
----------------------------
/sbin/rc1.d/S091vxvm-nodes-check[4]: /usr/sbin/lsdev: not found

VxVM general startup Dynamically linked vxconfigd not accessible
Output from "/sbin/rc1.d/S092vxvm-startup start":
----------------------------

Output from "/sbin/rc1.d/S093vxvm-reconfig start":
----------------------------
/sbin/rc1.d/S093vxvm-reconfig[48]: /etc/vx/lib/vxcommon: not found
"/sbin/rc1.d/S093vxvm-reconfig start" FAILED

The problem was first noticed when I applied some Veritas maintenance on the server and after it rebuilt the kernel and rebooted, the reboot failed with the same errors mentioned above:

vxfs mount; Cannot open /dev/vx/dsk/rootdg/tmpvol ; No such device or address

I received this on tmp, var, opt, usr, etc.....

I had to restore from my Ignite backup to recover the server. I am currently up and running without any problems but need to resolve what is causing this before re-trying to apply maintenance on this server again.

Re: VxVM startup errors in rc.log

Deoncia Grayson_1 — Mon, 13 Feb 2006 15:57:20 GMT

What Veritatas maint. did you install on your server?

Re: VxVM startup errors in rc.log

Steven E. Protter — Mon, 13 Feb 2006 15:57:33 GMT

Shalom,

You may wish to go back to an earlier ignite backup, because the one you used seems to have included this vxvm fault.

Thats my suggestion, also perhaps roll back to the previous kernel.

Have we possibly installed the LITS patch twice?

SEP

Re: VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 16:02:00 GMT

dr-x------ 3 root sys 96 Feb 13 16:01 PHCO_27100
dr-x------ 6 root sys 96 Feb 13 16:01 PHCO_27101
dr-x------ 3 root sys 96 Feb 13 16:01 PHCO_27103
dr-x------ 4 root sys 96 Feb 13 16:01 PHCO_31923
dr-x------ 4 root sys 96 Feb 13 16:01 PHCO_32202
dr-x------ 4 root sys 96 Feb 13 16:01 PHCO_32640
dr-x------ 5 root sys 96 Feb 13 16:01 PHCO_33310
dr-x------ 11 root sys 1024 Feb 13 16:01 PHCO_33533
dr-x------ 4 root sys 96 Feb 13 16:01 PHKL_23337
dr-x------ 4 root sys 96 Feb 13 16:01 PHKL_27321
dr-x------ 4 root sys 96 Feb 13 16:01 PHKL_28185
dr-x------ 6 root sys 96 Feb 13 16:01 PHKL_30607
dr-x------ 4 root sys 96 Feb 13 16:01 PHKL_33258
dr-x------ 6 root sys 96 Feb 13 16:01 PHKL_33390
dr-x------ 3 root sys 96 Feb 13 16:01 PHKL_33526
dr-x------ 6 root sys 96 Feb 13 16:01 PHNE_32616

Re: VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 16:06:52 GMT

The oldest rc.log I have is from a reboot of the server back on: Sun May 1 00:05:22 EDT 2005

and it has the same errors in the log:

VxVM device node check
Output from "/sbin/rc1.d/S091vxvm-nodes-check start":
----------------------------
/sbin/rc1.d/S091vxvm-nodes-check[4]: /usr/sbin/lsdev: not found

VxVM general startup Dynamically linked vxconfigd not accessible
Output from "/sbin/rc1.d/S092vxvm-startup start":
----------------------------

Output from "/sbin/rc1.d/S093vxvm-reconfig start":
----------------------------
/sbin/rc1.d/S093vxvm-reconfig[48]: /etc/vx/lib/vxcommon: not found
"/sbin/rc1.d/S093vxvm-reconfig start" FAILED

Re: VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 16:08:22 GMT

What is: LITS patch ??

Re: VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 16:18:12 GMT

My previous vmunix was from June 26, 2005. I just rebooted on that kernel and receive the same errors in rc.log and the startup puts out:

/sbin/fsclean: /dev/vx/dsk/rootdg/standvol (mounted) ok
HFS file systems are OK, not running fsck
Checking vxfs file systems
/dev/vx/dsk/rootdg/homevol :
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/homevol: No such device or address
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/homevol: No such device or address
file system check failure, aborting ...
/dev/vx/dsk/rootdg/tmpvol :
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/tmpvol: No such device or address
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/tmpvol: No such device or address
file system check failure, aborting ...
/dev/vx/dsk/rootdg/usrvol :
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/usrvol: No such device or address
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/usrvol: No such device or address
file system check failure, aborting ...
/dev/vx/dsk/rootdg/rootvol :
vxfs fsck: sanity check: root file system OK (mounted read/write)
/dev/vx/dsk/rootdg/optvol :
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/optvol: No such device or address
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/optvol: No such device or address
file system check failure, aborting ...
/dev/vx/dsk/rootdg/varvol :
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/varvol: No such device or address
vxfs fsck: Cannot open /dev/vx/dsk/rootdg/varvol: No such device or address
file system check failure, aborting ...

just as it does on the current kernel...

Re: VxVM startup errors in rc.log

Deoncia Grayson_1 — Mon, 13 Feb 2006 16:24:34 GMT

What is the latest patch bundle you've installed on your server?

Re: VxVM startup errors in rc.log

MikeL_4 — Mon, 13 Feb 2006 19:54:30 GMT

I currently have 35 HPUX servers al this same level and this is the only server we are having trouble with.

GOLDAPPS11i -> B.11.11.0506. Applications Patches for HP-UX 11i June 2005
GOLDBASE11i -> B.11.11.0506. Base Patches for HP-UX 11i v1, June 2005

The above mentioned patches I was trying to install are for Veritas VxFS Online maintenance recommended by HP. These same patches have already been installed on all of our HPUX servers except this server, it was the last to get them installed when it failed on reboot and I had to recover from Ignite backup.

Re: VxVM startup errors in rc.log

Sameer_Nirmal — Tue, 14 Feb 2006 00:06:41 GMT

Hi,

I am not sure what filesystem maintenance you did on the server in the single user mode. But it appears that the "/etc/inittab" file was replaced. With VXVM rootability, you need to have "/etc/initittab" with "vol1" and "vol2" entries for VXVM. You can cross check those entries with other working server having similar VXVM configuration.

Re: VxVM startup errors in rc.log

MikeL_4 — Tue, 14 Feb 2006 09:37:20 GMT

I was trying to get into single user mode as I needed to increase the size of a file system and we currently do not have OnLine VxFS installed, that was what my patches were for...

In any case, I checked our inittab file and these three lines were missing:
vxen::bootwait:/sbin/fs/vxfs/vxenablef -a
vol1::sysinit:/sbin/init.d/vxvm-sysboot /dev/console 2>&1 ##vxvm
vol2::sysinit:/sbin/init.d/vxvm-startup start /dev/console 2>&1 ##vxvm

I added them to the server, and rebooted and everything came up without any errors.

My oldest console log is from March 2005 and it showed these errors since then on reboots so I'm not sure when or how these entries were removed from inittab.

Thanks for your assistance...

Re: VxVM startup errors in rc.log

MikeL_4 — Tue, 14 Feb 2006 09:38:54 GMT

Thanks for your help....