topic Re: Wrong kernel booted? in Operating System - HP-UX

Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 05:54:26 GMT

Hi,

when I try to extend a mounted filesystem I get this

# fsadm -F vxfs -b $(lvdisplay $(bdf /opt|tail -1|cut -d\ -f1)|awk '/LV Size/{print$NF}')M /opt
fsadm: the -b option requires "HP OnLineJFS (Advanced Vxfs)"

although OnlineJFS should be available

# swlist -l fileset -a is_secure -a state -a title B3929BA
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/
#

# B3929BA HP OnLineJFS (Advanced VxFS)
# B3929BA.AdvJournalFS The Advanced VxFS File System
B3929BA.AdvJournalFS.AJFS-ENG-A-MAN AJFS-ENG-A-MAN true configured
B3929BA.AdvJournalFS.AJFS-JPN-E-MAN AJFS-JPN-E-MAN true configured
B3929BA.AdvJournalFS.AJFS-JPN-S-MAN AJFS-JPN-S-MAN true configured
B3929BA.AdvJournalFS.VXFS-ADV-KRN The Advanced VxFS File System Kernel files true confi
gured
B3929BA.AdvJournalFS.VXFS-ADV-RUN VXFS-ADV-RUN true configured

Could it be that I booted the wrong kernel that isn't patched to support this?
How can I check the running kernel's symbol table to see if it has compiled-in-support
(e.g. adb syntax)?

I tried this

# what /stand/vmunix|grep -i -e online -e adv -e vxfs
vxfs.c $Date: 1998/04/06 16:40:58 $Revision: r11ros/1 PATCH_11.00 (PHKL_12965)

but I don't even seem to have installed this very patch

# swlist -l product -a readme PHKL_12965|more
# Initializing...
# Contacting target "rustbin"...
ERROR: Software "PHKL_12965" was not found on host "rustbin:/".

Could this be the reason?

Rgds.
Ralph

Re: Wrong kernel booted?

Robert-Jan Goossens — Wed, 08 Dec 2004 06:11:56 GMT

H Ralph,

ask the kernel.

# echo 'boot_string/S' | adb /stand/vmunix /dev/mem

Regards,
Robert-Jan

Re: Wrong kernel booted?

Vishwas Pai — Wed, 08 Dec 2004 06:25:25 GMT

I had faced similar problem with JFS (though
I never used it). Initial install showed..

# mk_kernel -o /stand/vmunix
WARNING: No such device, tunable, or cdio vxadv.
Ignoring the following line.
vxadv

Then I saw vxadv driver was disabled in
/usr/conf/master.d (attached both origial &
fixed master files)

# mk_kernel -o /stand/vmunix
succeded without warning.

You should get something similar, when do a
nm on VMunix.

nm /stand/vmunix | grep -i vxad
vxadvc_link | 4653304|extern|entry |$CODE$

HTH -vishwas

p.s: I never used JFS, so I am not sure you
fsadm command will work.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 06:30:23 GMT

Robert-Jan,

yes, I already knew that it booted the kernel from the primary boot path as defined in IPL

# echo boot_string/S|adb /stand/vmunix /dev/kmem
Error from elf64_getehdr(application core file)
Not an Elf file: No Elf header
boot_string:
boot_string: disk(0/0/1/1.2.0.0.0.0.0;0)/stand/vmunix

# setboot
Primary bootpath : 0/0/1/1.2.0
Alternate bootpath : 0/0/2/0.2.0

Autoboot is ON (enabled)
Autosearch is OFF (disabled)

# lssf $(vgdisplay -v vg00|awk '/PV Name/{print$NF}')|awk '{print$(NF-1),$NF}'
0/0/1/1.2.0 /dev/dsk/c1t2d0
0/0/2/0.2.0 /dev/dsk/c2t2d0

Beneath /stand there isn't even any other kernel than the previous backup copy.
I rather would have expected some leftover prebuilt kernel in /stand/build which possibly wasn't moved because of a failing postbuild or postinstall or whatever *install script of the OnlineJFS bundle.

# find /stand -type f -name vmunix\* -exec ll {} \;
-rwxr-xr-x 1 root sys 19101032 Nov 30 08:04 /stand/vmunix
-rwxr-xr-x 1 root sys 19101040 Aug 3 16:33 /stand/vmunix.prev

But I cannot find any such script in the bundle anyway.

# swlist -l fileset -a postkernel -a postinstall -a unpostinstall -a preinstall B3929BA|grep -v ^#

B3929BA.AdvJournalFS.AJFS-ENG-A-MAN
B3929BA.AdvJournalFS.AJFS-JPN-E-MAN
B3929BA.AdvJournalFS.AJFS-JPN-S-MAN
B3929BA.AdvJournalFS.VXFS-ADV-KRN
B3929BA.AdvJournalFS.VXFS-ADV-RUN

No, what I meant was rather how to find out the driver capabilities of my kernel.

Re: Wrong kernel booted?

Robert-Jan Goossens — Wed, 08 Dec 2004 06:55:17 GMT

Hi Ralph,

You should have the Advanced VxFS File System Support in your kernel.

# strings /stand/vmunix | grep vxadv

Regards,
Robert-Jan

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 07:00:03 GMT

Nope, no hit

# strings /stand/vmunix|grep -c vxadv
0

but Vishwas made a good point.
If I check the vxfs driver's, what I would consider header file,
I can see various commented out lines regarding vxadv

# grep vxadv /usr/conf/master.d/vxfs
* Disabling AdvJournalFS: vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
* Disabling AdvJournalFS: vxadv vxbase
* Disabling AdvJournalFS: vxadv libvxfs_adv.a
* Disabling AdvJournalFS: vxadv

Re: Wrong kernel booted?

Vishwas Pai — Wed, 08 Dec 2004 07:12:58 GMT

I had attached the `vxfs' master.d file in my previous post (nothing but just uncommenting the driver configuration lines). I think you
should give it a try.

-vishwas

Re: Wrong kernel booted?

Cheryl Griffin — Wed, 08 Dec 2004 07:30:23 GMT

There is a group of patches that if installed prior to installing the AdvJFS product, the patches will disable the adv drivers so that if AdvJFS is later installed, you will need to manually enable the driver (regen the kernel, reboot) then select the patches again for install (not! reinstall) so that the AdvJFS portions of the patches can be installed.

This seems to be what has happened.

You need to install the latest version of check_patches (PHCO_27779) and look at the swagent.log to see what needs to be installed to fix this issue.

As a side note, if PHKL_18543 is not installed + dependencies as a minimum, you are behind on critical patches.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 08:03:17 GMT

Cheryl,

luckily I've already installed the patch you mentioned.

# swlist PHCO_27779
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/
#

# PHCO_27779 1.0 HP-UX Patch Tools and White Paper
PHCO_27779.RELEASE-NOTES 1.0 OS-Core.RELEASE-NOTES
PHCO_27779.SD-CMDS 1.0 SW-DIST.SD-CMDS

Yes, I can recall that after I had installed the core OS I did install MirrorDisk/UX and then the Quality Pack of March 2004.
It was only afterwards that I realized that also OnlineJFS needed to be installed,
what I did then, but of course forgetting to reinstall the missing vxadv patches from the Pack that were skipped in the first install.
So this must have caused my patch mismatch.

I now have run the check_patches script with the option to report for any missing patches.

# check_patches -m
Obtaining information on installed patches
Checking patch filesets for active patch 401 of 401
RESULT: Problems found, review /tmp/check_patches.report for details.

I attached the report from this run,
and yes, as you suspected, there are two patches for MirrorDisk/UX and OnlineJFS missing.
I'm confident that a simple swinstall of these two patches will create in a postprocess script run the kernel and dependencies automatically and attempt to boot the new kernel.

I will assign points as soon as the newly booted kernel will allow my fsadm command,
or come back to seek your help should have gone anything wrong.
I'm allowed to boot the box at 15:00 MEZ-1MESZ, which will be in about an hour.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 09:58:14 GMT

Meanwhile I could reinstall the two patches.

I searched for the Patch IDs of the two on the HP patch website, but also got selected a whole bunch of dependencies.

I downloaded the lot, and created the bundle with the acompanying script.

The bundle contains the following patches

# swlist -l fileset -s /var/spool/sw/VXADV_MIRRUX \*|grep ^#
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/var/spool/sw/VXADV_MIRRUX
#
# PHCO_21187 1.0 cumulative SAM/ObAM patch
# PHCO_23651 1.0 fsck_vxfs(1M) cumulative patch
# PHCO_27369 1.0 LVM commands (VA7410 HWE) cumulative patch
# PHKL_18543 1.0 PM/VM/UFS/async/scsi/io/DMAPI/JFS/perf patch
# PHKL_19202 1.0 fsadm panic if extending root on 11.x
# PHKL_20016 1.0 2nd CPU not recognized in G70/H70/I70
# PHKL_22589 1.0 LOFS, select(), IDS/9000 and umount race fix
# PHKL_27980 1.0 VxFS 3.1 cumulative patch: CR_EIEM
# PHKL_28766 1.0 Probe,IDDS,PM,VM,PA-8700,AIO,T600,FS,PDC,CLK
# PHKL_30553 1.0 LVM Cumulative Patch

Then I installed the bundle this way

# swinstall -v -x autoreboot=true -x patch_match_target=true -s /var/spool/sw/VXADV_MIRRUX \*

The autoreboot=true resulted in booting a new kernel, which however still seems to lack the desired functionality.

# fsadm -F vxfs -b 1520M /opt
fsadm: the -b option requires "HP OnLineJFS (Advanced Vxfs)"

At least did the swinstall mechanism yield in a new vxfs description file
(see, the commenting hashmarks disappeared)

# grep vxadv /usr/conf/master.d/vxfs
vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
vxadv vxbase
vxadv libvxfs_adv.a
vxadv

But grepping for some vxadv string in the kernel still fails.

# strings /stand/vmunix|grep -ci vxadv
0

# strings /stand/vmunix|grep -ci vxadv
0

The mtime of the kernel suggests a kernel rebuild has taken place.

# who -b; ll /stand/vmunix
. system boot Dec 8 15:18
-rwxr-xr-x 1 root sys 19103112 Dec 8 15:14 /stand/vmunix

Should manually rebuild the kernel anew?
What else can I do?

Re: Wrong kernel booted?

Dietmar Konermann — Wed, 08 Dec 2004 10:23:34 GMT

Ralph!

OK. Looks like vxadv is enabled in your master.d/vxfs file.

Should look like this now:

# grep vxadv /usr/conf/master.d/vxfs
vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
vxadv vxbase
vxadv libvxfs_adv.a
vxadv

Then your running kernel should have been configured with vxadv in the dfile/system file. Should look like this:

# echo "*system_data/s" | adb /stand/vmunix /dev/kmem | grep vx
vxadv
vxbase
vxportal

Now check if libvxadv.a is really linked to your kernel.

Compare the results of:

# what /usr/conf/lib/libvxfs_adv.a
/usr/conf/lib/libvxfs_adv.a:

with:

# what /stand/vmunix | grep -e vx_snap.c -e vx_reorg.c -e vx_kdmi.c -e vx_dio.c

Well. Let me know the results. :-)

Best regards...
Dietmar.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 10:40:26 GMT

Hi Dietmar,

first of all my kernel only has vx-related system_data symbols for vxbase support :-(

# echo \*system_data/s|adb /stand/vmunix /dev/kmem|grep vx
vxbase

The vxfs_adv library SCCS contents looks like this
(n.b. what does SCCS stand for, is it the Source Code Control System similar to RCS, or something completely different?)

# what /usr/conf/lib/libvxfs_adv.a
/usr/conf/lib/libvxfs_adv.a:
vx_kdmi.c $Date: 1998/11/03 08:00:12 $Revision: r11ros/cup_ros_ep1_pb/1 PATCH_11.00 (PHKL_1681
9)
vx_dio.c $Date: 1999/05/10 00:38:37 $Revision: r11ros/cup_ros_ep3_pb/2 PATCH_11.00 (PHKL_18543
)

while there aren't any matches for the wanted source files.

# what /stand/vmunix|grep -cE 'vx_(snap|reorg|kdmi|dio).c'
0

Re: Wrong kernel booted?

Tom Danzig — Wed, 08 Dec 2004 10:43:20 GMT

Just a thought ... is your OnlineJFS license valid? You can check with:

# /sbin/vxlicense -p

That should print out your license info. Perhaps something went wrong with your license key during the install.

Re: Wrong kernel booted?

Dietmar Konermann — Wed, 08 Dec 2004 10:44:34 GMT

Well, Ralph!

Essentially system_data is a 1:1 copy of the system file that was used with mk_kernel. So please the /stand/system for vxadv.

Then build a new kernel with a system file containing vxadv and boot that kernel.

Best regards...
Dietmar.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 10:52:41 GMT

Tom,

I can't find the command you cited.
All sort of vx* commands I've got in /sbin are those

# echo /sbin/vx* /usr/sbin/vx*
/sbin/vxdump /sbin/vxrestore /sbin/vxupgrade /usr/sbin/vxdump /usr/sbin/vxrestore

However, I know of a different Veritas command to report license info on our FSC Solaris boxes.
There exists such a command

# uname -srv;/sbin/vxlicrep -s|grep Product\ Name
SunOS 5.8 Generic_108528-27
Product Name = VERITAS Foundation Suite
Product Name = VERITAS Volume Manager
Product Name = VERITAS File System
Product Name = VERITAS SANPoint Control
Product Name = VERITAS Foundation Suite
Product Name = VERITAS Volume Manager
Product Name = VERITAS File System
Product Name = VERITAS SANPoint Control
Product Name = VERITAS Cluster Server

Re: Wrong kernel booted?

Dietmar Konermann — Wed, 08 Dec 2004 10:55:21 GMT

Don't search for vxlicense on your 11.00 server with VxFS 3.1 only. :-) You would need to install VxFS 3.3 or other Veritas products to get that onto the system.

Re: Wrong kernel booted?

Tom Danzig — Wed, 08 Dec 2004 10:55:41 GMT

/sbin/vxlicense is a symlink to /sbin/fs/vxfs3.3/vxlicense

Re: Wrong kernel booted?

Tom Danzig — Wed, 08 Dec 2004 10:58:41 GMT

Ignore my ramblings and listen to Dietmar.

Re: Wrong kernel booted?

Ralph Grothe — Wed, 08 Dec 2004 11:10:51 GMT

Dietmar,

I'm a bit in a hurry to make use of the maintenance window for servers of this stack, while I can reboot them.
Since I usually do kernel tuning on HP-UX with SAM (one of the few things I ever use it for) I lack practice in doing it manually.
Could you provide me with a short algorithm how to make use of kmsystem and the mk_kernel script to this end.
Unfortunately I lack time for browsing through the Cookbook or havin a look at the script right now.

Re: Wrong kernel booted?

Dietmar Konermann — Wed, 08 Dec 2004 11:16:21 GMT

If you need to be fast:

# kmsystem -c y vxadv
# mk_kernel
# kmupdate
# /sbin/init.d/kmbuild stop
# reboot

Best regards...
Dietmar.