Wrong kernel booted?

[Ralph Grothe]

Hi,

when I try to extend a mounted filesystem I get this

# fsadm -F vxfs -b $(lvdisplay $(bdf /opt|tail -1|cut -d\ -f1)|awk '/LV Size/{print$NF}')M /opt
fsadm: the -b option requires "HP OnLineJFS (Advanced Vxfs)"


although OnlineJFS should be available

# swlist -l fileset -a is_secure -a state -a title B3929BA
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/
#

# B3929BA HP OnLineJFS (Advanced VxFS)
# B3929BA.AdvJournalFS The Advanced VxFS File System
B3929BA.AdvJournalFS.AJFS-ENG-A-MAN AJFS-ENG-A-MAN true configured
B3929BA.AdvJournalFS.AJFS-JPN-E-MAN AJFS-JPN-E-MAN true configured
B3929BA.AdvJournalFS.AJFS-JPN-S-MAN AJFS-JPN-S-MAN true configured
B3929BA.AdvJournalFS.VXFS-ADV-KRN The Advanced VxFS File System Kernel files true confi
gured
B3929BA.AdvJournalFS.VXFS-ADV-RUN VXFS-ADV-RUN true configured


Could it be that I booted the wrong kernel that isn't patched to support this?
How can I check the running kernel's symbol table to see if it has compiled-in-support
(e.g. adb syntax)?

I tried this

# what /stand/vmunix|grep -i -e online -e adv -e vxfs
vxfs.c $Date: 1998/04/06 16:40:58 $Revision: r11ros/1 PATCH_11.00 (PHKL_12965)

but I don't even seem to have installed this very patch

# swlist -l product -a readme PHKL_12965|more
# Initializing...
# Contacting target "rustbin"...
ERROR: Software "PHKL_12965" was not found on host "rustbin:/".

Could this be the reason?

Rgds.
Ralph

[Robert-Jan Goossens]

H Ralph,

ask the kernel.

# echo 'boot_string/S' | adb /stand/vmunix /dev/mem

Regards,
Robert-Jan

[Vishwas Pai]

I had faced similar problem with JFS (though
I never used it). Initial install showed..

# mk_kernel -o /stand/vmunix
WARNING: No such device, tunable, or cdio vxadv.
Ignoring the following line.
vxadv

Then I saw vxadv driver was disabled in
/usr/conf/master.d (attached both origial &
fixed master files)

# mk_kernel -o /stand/vmunix
succeded without warning.

You should get something similar, when do a
nm on VMunix.

nm /stand/vmunix | grep -i vxad
vxadvc_link | 4653304|extern|entry |$CODE$

HTH -vishwas

p.s: I never used JFS, so I am not sure you
fsadm command will work.

[Ralph Grothe]

Robert-Jan,

yes, I already knew that it booted the kernel from the primary boot path as defined in IPL

# echo boot_string/S|adb /stand/vmunix /dev/kmem
Error from elf64_getehdr(application core file)
Not an Elf file: No Elf header
boot_string:
boot_string: disk(0/0/1/1.2.0.0.0.0.0;0)/stand/vmunix

# setboot
Primary bootpath : 0/0/1/1.2.0
Alternate bootpath : 0/0/2/0.2.0

Autoboot is ON (enabled)
Autosearch is OFF (disabled)

# lssf $(vgdisplay -v vg00|awk '/PV Name/{print$NF}')|awk '{print$(NF-1),$NF}'
0/0/1/1.2.0 /dev/dsk/c1t2d0
0/0/2/0.2.0 /dev/dsk/c2t2d0


Beneath /stand there isn't even any other kernel than the previous backup copy.
I rather would have expected some leftover prebuilt kernel in /stand/build which possibly wasn't moved because of a failing postbuild or postinstall or whatever *install script of the OnlineJFS bundle.

# find /stand -type f -name vmunix\* -exec ll {} \;
-rwxr-xr-x 1 root sys 19101032 Nov 30 08:04 /stand/vmunix
-rwxr-xr-x 1 root sys 19101040 Aug 3 16:33 /stand/vmunix.prev


But I cannot find any such script in the bundle anyway.


# swlist -l fileset -a postkernel -a postinstall -a unpostinstall -a preinstall B3929BA|grep -v ^#

B3929BA.AdvJournalFS.AJFS-ENG-A-MAN
B3929BA.AdvJournalFS.AJFS-JPN-E-MAN
B3929BA.AdvJournalFS.AJFS-JPN-S-MAN
B3929BA.AdvJournalFS.VXFS-ADV-KRN
B3929BA.AdvJournalFS.VXFS-ADV-RUN


No, what I meant was rather how to find out the driver capabilities of my kernel.

[Robert-Jan Goossens]

Hi Ralph,

You should have the Advanced VxFS File System Support in your kernel.

# strings /stand/vmunix | grep vxadv

Regards,
Robert-Jan

[Ralph Grothe]

Nope, no hit

# strings /stand/vmunix|grep -c vxadv
0


but Vishwas made a good point.
If I check the vxfs driver's, what I would consider header file,
I can see various commented out lines regarding vxadv


# grep vxadv /usr/conf/master.d/vxfs
* Disabling AdvJournalFS: vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
* Disabling AdvJournalFS: vxadv vxbase
* Disabling AdvJournalFS: vxadv libvxfs_adv.a
* Disabling AdvJournalFS: vxadv

[Vishwas Pai]

I had attached the `vxfs' master.d file in my previous post (nothing but just uncommenting the driver configuration lines). I think you
should give it a try.

-vishwas

[Cheryl Griffin · Accepted solution]

There is a group of patches that if installed prior to installing the AdvJFS product, the patches will disable the adv drivers so that if AdvJFS is later installed, you will need to manually enable the driver (regen the kernel, reboot) then select the patches again for install (not! reinstall) so that the AdvJFS portions of the patches can be installed.

This seems to be what has happened.

You need to install the latest version of check_patches (PHCO_27779) and look at the swagent.log to see what needs to be installed to fix this issue.

As a side note, if PHKL_18543 is not installed + dependencies as a minimum, you are behind on critical patches.

[Ralph Grothe]

Cheryl,

luckily I've already installed the patch you mentioned.

# swlist PHCO_27779
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/
#

# PHCO_27779 1.0 HP-UX Patch Tools and White Paper
PHCO_27779.RELEASE-NOTES 1.0 OS-Core.RELEASE-NOTES
PHCO_27779.SD-CMDS 1.0 SW-DIST.SD-CMDS


Yes, I can recall that after I had installed the core OS I did install MirrorDisk/UX and then the Quality Pack of March 2004.
It was only afterwards that I realized that also OnlineJFS needed to be installed,
what I did then, but of course forgetting to reinstall the missing vxadv patches from the Pack that were skipped in the first install.
So this must have caused my patch mismatch.

I now have run the check_patches script with the option to report for any missing patches.

# check_patches -m
Obtaining information on installed patches
Checking patch filesets for active patch 401 of 401
RESULT: Problems found, review /tmp/check_patches.report for details.


I attached the report from this run,
and yes, as you suspected, there are two patches for MirrorDisk/UX and OnlineJFS missing.
I'm confident that a simple swinstall of these two patches will create in a postprocess script run the kernel and dependencies automatically and attempt to boot the new kernel.

I will assign points as soon as the newly booted kernel will allow my fsadm command,
or come back to seek your help should have gone anything wrong.
I'm allowed to boot the box at 15:00 MEZ-1MESZ, which will be in about an hour.

[Ralph Grothe]

Meanwhile I could reinstall the two patches.

I searched for the Patch IDs of the two on the HP patch website, but also got selected a whole bunch of dependencies.

I downloaded the lot, and created the bundle with the acompanying script.

The bundle contains the following patches

# swlist -l fileset -s /var/spool/sw/VXADV_MIRRUX \*|grep ^#
# Initializing...
# Contacting target "rustbin"...
#
# Target: rustbin:/var/spool/sw/VXADV_MIRRUX
#
# PHCO_21187 1.0 cumulative SAM/ObAM patch
# PHCO_23651 1.0 fsck_vxfs(1M) cumulative patch
# PHCO_27369 1.0 LVM commands (VA7410 HWE) cumulative patch
# PHKL_18543 1.0 PM/VM/UFS/async/scsi/io/DMAPI/JFS/perf patch
# PHKL_19202 1.0 fsadm panic if extending root on 11.x
# PHKL_20016 1.0 2nd CPU not recognized in G70/H70/I70
# PHKL_22589 1.0 LOFS, select(), IDS/9000 and umount race fix
# PHKL_27980 1.0 VxFS 3.1 cumulative patch: CR_EIEM
# PHKL_28766 1.0 Probe,IDDS,PM,VM,PA-8700,AIO,T600,FS,PDC,CLK
# PHKL_30553 1.0 LVM Cumulative Patch


Then I installed the bundle this way

# swinstall -v -x autoreboot=true -x patch_match_target=true -s /var/spool/sw/VXADV_MIRRUX \*


The autoreboot=true resulted in booting a new kernel, which however still seems to lack the desired functionality.

# fsadm -F vxfs -b 1520M /opt
fsadm: the -b option requires "HP OnLineJFS (Advanced Vxfs)"


At least did the swinstall mechanism yield in a new vxfs description file
(see, the commenting hashmarks disappeared)

# grep vxadv /usr/conf/master.d/vxfs
vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
vxadv vxbase
vxadv libvxfs_adv.a
vxadv


But grepping for some vxadv string in the kernel still fails.

# strings /stand/vmunix|grep -ci vxadv
0

# strings /stand/vmunix|grep -ci vxadv
0

The mtime of the kernel suggests a kernel rebuild has taken place.

# who -b; ll /stand/vmunix
. system boot Dec 8 15:18
-rwxr-xr-x 1 root sys 19103112 Dec 8 15:14 /stand/vmunix


Should manually rebuild the kernel anew?
What else can I do?

[Dietmar Konermann]

Ralph!

OK. Looks like vxadv is enabled in your master.d/vxfs file.

Should look like this now:

# grep vxadv /usr/conf/master.d/vxfs
vxadv vxadvc 10 100 -1 -1
dmapi vxadv vxbase
vxportal vxadv vxbase
vxadv vxbase
vxadv libvxfs_adv.a
vxadv

Then your running kernel should have been configured with vxadv in the dfile/system file. Should look like this:

# echo "*system_data/s" | adb /stand/vmunix /dev/kmem | grep vx
vxadv
vxbase
vxportal

Now check if libvxadv.a is really linked to your kernel.

Compare the results of:

# what /usr/conf/lib/libvxfs_adv.a
/usr/conf/lib/libvxfs_adv.a:

with:

# what /stand/vmunix | grep -e vx_snap.c -e vx_reorg.c -e vx_kdmi.c -e vx_dio.c

Well. Let me know the results. :-)

Best regards...
Dietmar.

[Ralph Grothe]

Hi Dietmar,

first of all my kernel only has vx-related system_data symbols for vxbase support :-(

# echo \*system_data/s|adb /stand/vmunix /dev/kmem|grep vx
vxbase


The vxfs_adv library SCCS contents looks like this
(n.b. what does SCCS stand for, is it the Source Code Control System similar to RCS, or something completely different?)


# what /usr/conf/lib/libvxfs_adv.a
/usr/conf/lib/libvxfs_adv.a:
vx_kdmi.c $Date: 1998/11/03 08:00:12 $Revision: r11ros/cup_ros_ep1_pb/1 PATCH_11.00 (PHKL_1681
9)
vx_dio.c $Date: 1999/05/10 00:38:37 $Revision: r11ros/cup_ros_ep3_pb/2 PATCH_11.00 (PHKL_18543
)


while there aren't any matches for the wanted source files.

# what /stand/vmunix|grep -cE 'vx_(snap|reorg|kdmi|dio).c'
0

[Tom Danzig]

Just a thought ... is your OnlineJFS license valid? You can check with:

# /sbin/vxlicense -p

That should print out your license info. Perhaps something went wrong with your license key during the install.

[Dietmar Konermann]

Well, Ralph!

Essentially system_data is a 1:1 copy of the system file that was used with mk_kernel. So please the /stand/system for vxadv.

Then build a new kernel with a system file containing vxadv and boot that kernel.

Best regards...
Dietmar.

[Ralph Grothe]

Tom,

I can't find the command you cited.
All sort of vx* commands I've got in /sbin are those

# echo /sbin/vx* /usr/sbin/vx*
/sbin/vxdump /sbin/vxrestore /sbin/vxupgrade /usr/sbin/vxdump /usr/sbin/vxrestore


However, I know of a different Veritas command to report license info on our FSC Solaris boxes.
There exists such a command

# uname -srv;/sbin/vxlicrep -s|grep Product\ Name
SunOS 5.8 Generic_108528-27
Product Name = VERITAS Foundation Suite
Product Name = VERITAS Volume Manager
Product Name = VERITAS File System
Product Name = VERITAS SANPoint Control
Product Name = VERITAS Foundation Suite
Product Name = VERITAS Volume Manager
Product Name = VERITAS File System
Product Name = VERITAS SANPoint Control
Product Name = VERITAS Cluster Server

[Dietmar Konermann]

Don't search for vxlicense on your 11.00 server with VxFS 3.1 only. :-) You would need to install VxFS 3.3 or other Veritas products to get that onto the system.

[Tom Danzig]

/sbin/vxlicense is a symlink to /sbin/fs/vxfs3.3/vxlicense

[Tom Danzig]

Ignore my ramblings and listen to Dietmar.

[Ralph Grothe]

Dietmar,

I'm a bit in a hurry to make use of the maintenance window for servers of this stack, while I can reboot them.
Since I usually do kernel tuning on HP-UX with SAM (one of the few things I ever use it for) I lack practice in doing it manually.
Could you provide me with a short algorithm how to make use of kmsystem and the mk_kernel script to this end.
Unfortunately I lack time for browsing through the Cookbook or havin a look at the script right now.

[Dietmar Konermann]

If you need to be fast:

# kmsystem -c y vxadv
# mk_kernel
# kmupdate
# /sbin/init.d/kmbuild stop
# reboot

Best regards...
Dietmar.

[Ralph Grothe]

Great,

# kmsystem -c y vxadv

# grep vx /stand/system
vxadv
vxbase

# mk_kernel -v -s /stand/system
Building a new kernel based on template file "/stand/system"
Generating module: krm...
Compiling /stand/build/conf.c...
Loading the kernel...
Generating kernel symbol table...

# strings /stand/build/vmunix_test|grep -i vxadv
vxadv
vxadv

:-)

# kmupdate

Kernel update request is scheduled.

Default kernel /stand/vmunix will be updated by
newly built kernel /stand/build/vmunix_test
at next system shutdown or startup time.


# ll /stand/.update_ux && cat /stand/.update_ux
-rw-r--r-- 1 root sys 25 Dec 8 17:24 /stand/.update_ux
/stand/build/vmunix_test

# cd / && /sbin/shutdown -ry 0

[Dietmar Konermann]

Well, looks great. Let's wait and see. :-)

[Ralph Grothe]

Since I already extended the FS offline,
I will try Advanced VxFS functionality of the kernel differently.

This needed to be extended to host the Java 1.4.2 monster.

# bdf /opt
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol6 2048000 1608008 436576 79% /opt

# lvcreate -L 96 -n lvsnap vg00
Logical volume "/dev/vg00/lvsnap" has been successfully created with
character device "/dev/vg00/rlvsnap".
Logical volume "/dev/vg00/lvsnap" has been successfully extended.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf

# mkdir -p /mnt/snap

# mount -F vxfs -o snapof=/opt /dev/vg00/lvsnap /mnt/snap

# bdf -i /opt /mnt/snap
Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol6 2048000 1608008 436576 79% 30951 13721 69% /opt
/dev/vg00/lvsnap 2048000 1608008 436560 79% 31008 13728 69% /mnt/snap

Hurray, my kernel is vxadv aware :-)

Many thanks for your help!