topic Re: System won't boot after kernel parameter change in Operating System - HP-UX

System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 09:36:07 GMT

I changed the kernel parameter maxfiles and maxfiles_lim to 409600. When I try to reboot the server, it won't boot anymore. Do you have any idea on how to recover my vpar?

Thanks.

Re: System won't boot after kernel parameter change

sujit kumar singh — Tue, 03 Feb 2009 09:43:45 GMT

hi

yiu can try boooting the alternate kernel that is /stand/vmunix.old

from the MON prompt try the following

MON>vparload -p -b

the alternate kernel name may be /stand/vmunix.old

If you had another Boot Disk in the VPAR which you might have detached during this kernel change, you can put in that disk which has the old kernel and try from the MON prompt

MON>vparload -p -B

Regards
Sujit

Regards
Sujit

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 10:02:16 GMT

Tried it. System still won't boot. Even when I use vmunix.prev

Any other ideas?

Thanks.

Re: System won't boot after kernel parameter change

sujit kumar singh — Tue, 03 Feb 2009 10:05:26 GMT

What is the Error that you are getting trying to boot from the alternate Kernel and what is the error that is for booting the Old kernel?

can you please post the errors in both the cases please?

Regards
Sujit

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 11:29:29 GMT

Error is the same as booting from the old and new kernels. Shown below:

WARNING: Could not allocate page table structures in

interleaved memory.
WARNING: For better performance, it is recommended that you

re-configure
WARNING: the machine to have more interleaved meError:vPar

shmem r/o private for type 4 is not yet ready.Try later
Error:vPar shmem r/o private for type 4 is not yet ready.Try

later

Stored message buffer up to panic:
Found adjacent data tr. Growing size. 0x3179000 ->

0x7179000.
Pinned PDK malloc pool: base: 0xe000000100e87000 size=116196K
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.

MFS is defined: base= 0xe000000100e87000 size= 5696 KB

WARNING: Could not allocate page table structures in

interleaved memory.
WARNING: For better performance, it is recommended that you

re-configure

System Panic:
WARNING: the machine to have more interleaved memory.
panic: Could not find a good physical range for unpinned space
Stack Trace:
IP Function Name
0xe000000000e64e90 unpinned_pdk_malloc_phys_mem_alloc+0x3c0
0xe000000000e65de0 unpinned_pdk_malloc_init+0x6d0
0xe0000000017dee50 DoCalllist+0x3a0
End of Stack Trace

linkstamp: Fri Jan 30 12:11:41 EAT 2009
_release_version: @(#) $Revision: vmunix: B11.23_LR

FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
Error:vPar shmem r/o private for type 4 is not yet ready.Try

later
Error:vPar shmem r/o private for type 4 is not yet ready.Try

later

*** A system crash has occurred. (See the above messages for

details.)
*** The system is now preparing to dump physical memory to

disk, for use
*** in debugging the crash.

ERROR: Your system crashed before I/O and dump configuration

was complete.
This system does not support a crash dump under these

circumstances.
Contact your HP support representative for assistance.
Initializing IO Devices ...
LBA Cell 00 (10/11): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card

==============================================================

======
10 02 0x00 0xa9 0x01 0x00 PCI Bridge

(0x01a7,0x1014)
10 02 0x00 0xaa 0x04 0x00 SCSI Adapter

(0x0030,0x1000)
10 02 0x00 0xaa 0x04 0x01 SCSI Adapter

(0x0030,0x1000)
10 02 0x00 0xaa 0x06 0x00 Ethernet

(0x1079,0x8086)
10 02 0x00 0xaa 0x06 0x01 Ethernet

(0x1079,0x8086)
LBA Cell 00 (04/05): Occupied PCI-X 266MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card

==============================================================

======
04 06 0x00 0x54 0x01 0x00 Fibre Channel

(0x2422,0x1077)
04 06 0x00 0x54 0x01 0x01 Fibre Channel

(0x2422,0x1077)
LBA Cell 01 (14/15): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card

==============================================================

======
14 04 0x01 0xe3 0x01 0x00 Ethernet

(0x1079,0x8086)
14 04 0x01 0xe3 0x01 0x01 Ethernet

(0x1079,0x8086)
LBA Cell 01 (02/03): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card

==============================================================

======
02 07 0x01 0x38 0x01 0x00 Fibre Channel

(0x2422,0x1077)
02 07 0x01 0x38 0x01 0x01 Fibre Channel

(0x2422,0x1077)
Complete

[MON]
vPar "ANALYTICS" failed to load the kernel due to:

non-bootable boot media; incorrect vPars software installed;

or incorrect boot media EFI path in vPars database.
Try using "vparload -p ANALYTICS -E 0" to load the vPar.
Error loading "ANALYTICS". Shutting down "ANALYTICS"...
[MON] ANALYTICS has halted.

We already tried vparload -p ANALYTICS -E 0, still doesn't work.

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 12:03:31 GMT

First, halt the vPar (should halt itself eventually, you might have to do a vparreset from another vPar, though).

Then try one of the following:

1) vparload -p -b /stand/backup/vmunix

[vmunix.prev is more of an 11.11-ism. For v2 and later, you specify the backup/alternate kernel configuration -- the default auto backup is "backup" so this is likely to be there for your pre-tunable changes].

2) vparload -p -o \"maxfiles=2048 maxfiles_lim=4096\"

That may only work on a v3 kernel -- I don't remember for sure when boot-string tunable settings was done

3) Add more interleave to the vPar until it boots. What the problem is here is that you've increased this tunable -- and kernel data structures which use it and are allocated relatively early in boot grew accordingly. The problem is that early boot memory is a very scarce resource (there's a bootstrapping going on -- a little known good memory is used to get things built to bring in some more memory... then that memory is used to discover the rest of the system and bring the rest of the memory in, after which things work pretty normally from the allocator's point of view. Unpinned PDK memory is that second bit [Pinned is the original known good if you're wondering]). What you've indirectly asked the system to build is just too much for the memory known/available at this intermediate stage (which is usually just Interleave [or on a system with only CLM, the CLM which is addressed from low ranges].)

Personally, I'd go option (1) and then re-tune the system.

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 12:28:31 GMT

Tried options 1 and 2 to no avail. How do I add more interleaves? Maybe that will work. Thanks.

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 14:43:28 GMT

What happened with (1)? Was there no backup configuration on your machine [you could try last_install as well... but it would be nice to get some data on what your output is here].

Oh -- and one other thing to try:

vparload -p -o -tm

I had to dig around to find Failsafe mode -- but that should bring up the kernel with your tunable settings ignored and the Failsafe values instead. Try that before adding memory as described below. (And if that doesn't work, show what did happen -- getting the options recognized by vparload sometimes seems to involve getting the command to realize what goes with -o and what doesn't... so it may be that we just need to get a "" around the -tm or something.

You add memory to a vPar using vparmodify. Since this is v2, the vPar will have to be stopped (which shouldn't be an issue).

vparmodify -p -a mem::

Where vpar is the target vPar and size is the amount in Mb you want to add [since you don't specify a Cell, that is ILM in the command]. The size needs to be a multiple of the granule size defined for your vPar environment.

vparstatus -A in another vPar will show you available resources (granule size and how much ILM is left).

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 15:09:15 GMT

Thanks. Succeeded in booting in failsafe mode. The OS advised us to removed the saved with the kconfig -d command but when we issued it, the OS returned that the command was not found. How do we reset the kernel parameters to the old values in failsafe mode. Please advise. Thanks again!

OWEN

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 15:25:51 GMT

Ok... failsafe comes up in Single user mode.

So you'll want to "mount /usr" first [as kconfig lives in /usr/sbin/kconfig].

Then I'd do just "kconfig" first -- to see which configurations you _do_ have. Look for one such as "GOOD" or "backup" or anything other than your current one. Then kconfig -n and reboot.

Your system should come up in _that_ configuration -- at which point you can do: kctune -c maxfiles= maxfiles_lim=
which will reset the tunables to the default for that configuration. Then kconfig -n and reboot... and you should be back to normal. Alternately, you could try a lower setting than 409600 but higher than the defaults -- but at this point I'd be more focused on getting your system back.

After you mount /usr -- you could try just kctune directly... but my v3 system in Failsafe has a long message about configuration changes not likely to be written to disk in failsafe mode, so I'd assume v2 is the same here. Safer to reboot to an alternate configuration (non-Failsafe) and make the changes there.

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 15:44:49 GMT

Thanks Don,

However, when we tried to mount /usr, the system returned that /usr is busy. Is there another way around this? Booted the vpar in LVM maintenance mode but the system would not boot also.

Owen

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 16:27:22 GMT

Do a "ls /stand" and note the directory names. Those are the configurations -- and you'd be looking for anything other than current, I would think. Assuming you find one, go back to "vparload -p -b /stand//vmunix" -- that should boot the alternate configuration with prior tunable values. At which point, you can kctune -c the other configuration to get back.

last_install would be a good one if it is there too.

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 16:29:00 GMT

We tried to boot from the last_install. Now, the kernel won't even boot in failsafe mode.

Owen

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 16:33:52 GMT

All with the same out-of-memory panic? (Which wouldn't make any sense unless last_install is the only real configuration... but that's not at all the default).

Console logs of the vPar booting from loading vmunix to the panic might help here.

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 16:36:31 GMT

Here's what the console shows:

Initializing IO Devices ...
LBA Cell 00 (10/11): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card
====================================================================
10 02 0x00 0xa9 0x01 0x00 PCI Bridge (0x01a7,0x1014)
10 02 0x00 0xaa 0x04 0x00 SCSI Adapter (0x0030,0x1000)
10 02 0x00 0xaa 0x04 0x01 SCSI Adapter (0x0030,0x1000)
10 02 0x00 0xaa 0x06 0x00 Ethernet (0x1079,0x8086)
10 02 0x00 0xaa 0x06 0x01 Ethernet (0x1079,0x8086)
LBA Cell 00 (04/05): Occupied PCI-X 266MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card
====================================================================
04 06 0x00 0x54 0x01 0x00 Fibre Channel (0x2422,0x1077)
04 06 0x00 0x54 0x01 0x01 Fibre Channel (0x2422,0x1077)
LBA Cell 01 (14/15): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card
====================================================================
14 04 0x01 0xe3 0x01 0x00 Ethernet (0x1079,0x8086)
14 04 0x01 0xe3 0x01 0x01 Ethernet (0x1079,0x8086)
LBA Cell 01 (02/03): Occupied PCI-X 133MHz
Scan PCI:
Rope Slot Seg Bus Dev Fun Card
====================================================================
02 07 0x01 0x38 0x01 0x00 Fibre Channel (0x2422,0x1077)
02 07 0x01 0x38 0x01 0x01 Fibre Channel (0x2422,0x1077)
Complete
Loading Clients ...
Loading SAL_ABI ... (text) (data) Complete
Loading ACPI ...... (text) (data) Complete
Loading EFI ...... (text) (data) Complete
12125503 bytes free left in the Pal/Client block
Starting Clients ...
Initializing ACPI tables ...
Initializing FACS table (2ef470028/40) Complete
Initializing DBGP table (2ef470068/34) Complete
Initializing HPET table (2ef4700a0/38) Complete
Initializing GSP table (2ef4700d8/74) Complete
Initializing MADT table (2ef470150/1fc) Complete
Initializing DSDT table (2ef470350/3336b) Complete
Initializing SPMI table (2ef4a36c0/50) Complete
Initializing FADT table (2ef4a3710/f4) Complete
Initializing SLIT table (2ef4a3808/35) Complete
Initializing CPEP table (2ef4a3840/8c) Complete
Initializing SRAT table (2ef4a38d0/1f90) Complete
Initializing HCDP table (2ef4a5860/44) Complete
Initializing SBCT table (2ef4a58a8/30) Complete

Initializing XSDT table (2ef4a58d8/ec) Complete
Initializing SSDT table 00 (2ef4a59c8/23cc6) Complete
Initializing RSDP table (2ef470000/24) Complete
ACPI Initialization complete: table size = 366224 / 7995392 (4%)

Initializing SAL_ABI ...
Assigning PDT to current fpar ... Complete
Processing firmware memory tables ... Complete
Relocating PAL ... Complete
Processing platform I/O information ... Complete
Building ESI Table ... Complete
Loading ACPI tables ... Complete
Building EFI memory table ... Complete
Building PCI cache structure ... Complete

Welcome to EFI 2.0!
EFI version 2.00 [14.62] Running on GenuineIntel Itanium(R) Processor
EFI server rx7640 SAL [12:58:14] - HP
Loading 'BRM5701_LAN'...

Copyright (c) 2000-2005 Broadcom Corporation
Broadcom NetXtreme Gigabit Ethernet EFI driver v7.0.11

driver: Image BRM5701_LAN loaded at 2ECE44000-2ECE7E000. Returned Success
Loading 'FPSWA'...
driver: Image FPSWA loaded at 2ECDD8000-2ECE42000. Returned Success
PciRom S=01 B=38 D=1 F=0 I=3 loaded at 5FBFCD000-5FC000000
HP PCI-X 2Port 4Gb Fibre Channel Adapter (driver 1.09, firmware 4.00.090)
HP PCI-X 2Port 4Gb Fibre Channel Adapter (driver 1.09, firmware 4.00.090)
PciRom S=00 B=AA D=4 F=0 I=0 loaded at 5FBF68000-5FBFCE000
PciRom S=00 B=AA D=6 F=0 I=0 loaded at 2ECC76000-2ECDD6000
Scsi(Pun0,Lun0) COMPAQ BF1468B26B HPB9 (320 MBytes/sec)
Scsi(Pun1,Lun0) COMPAQ BF1468B26B HPB9 (320 MBytes/sec)
POSSE Library is loading...
EFI Shell UEFI version 2.00 [14.62]
Device mapping table
fs0 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)/HD(Part1,Sig13168F0C-ECDD-11DD-8002-D6217B60E588)
fs1 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)/HD(Part3,Sig13168F3E-ECDD-11DD-8004-D6217B60E588)
fs2 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun1,Lun0)/HD(Part1,Sig202B74E2-ED04-11DD-8002-D6217B60E588)
blk0 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)
blk1 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)/HD(Part1,Sig13168F0C-ECDD-11DD-8002-D6217B60E588)
blk2 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)/HD(Part2,Sig13168F2A-ECDD-11DD-8003-D6217B60E588)
blk3 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun0,Lun0)/HD(Part3,Sig13168F3E-ECDD-11DD-8004-D6217B60E588)
blk4 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun1,Lun0)
blk5 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun1,Lun0)/HD(Part1,Sig202B74E2-ED04-11DD-8002-D6217B60E588)
blk6 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun1,Lun0)/HD(Part2,Sig202B7500-ED04-11DD-8003-D6217B60E588)
blk7 : Acpi(HWP0002,PNP0A03,A)/Pci(1|0)/Pci(4|0)/Scsi(Pun1,Lun0)/HD(Part3,Sig202B751E-ED04-11DD-8004-D6217B60E588)
EFI Boot Manager ver 2.00 [14.62] Loading device drivers
EFI Boot Manager ver 2.00 [14.62] Please select a boot option
OS might use only the primary console set via boot manager or conconfig command

0/0/10/1/0/4/0.0.0.0.0.0.0
EFI Shell [Built*in]
0/0/10/1/0.4.0.1.0
Boot Option Maintenance Menu

Use ^ and v to change option(s). Use Enter to select an option
Use > and < to view long descriptions
Hot Key: 'S' or 's' to start the EFI Shell

Selected boot option: 0/0/10/1/0/4/0.0.0.0.0.0.0

Loading.: 0/0/10/1/0/4/0.0.0.0.0.0.0

Executing Image: "\EFI\HPUX\HPUX.EFI"
Optional Data : "\EFI\HPUX\HPUX.EFI boot /stand/vmunix -tm"

Starting: 0/0/10/1/0/4/0.0.0.0.0.0.0

(C) Copyright 1999-2006 Hewlett-Packard Development Company, L.P.
All rights reserved

HP-UX Boot Loader for IPF -- Revision 2.029

Error: kernel does not contain required vPars components.

[MON]
vPar "ANALYTICS" failed to load the kernel for either trying to boot an unsupported kernel in vPars environment or having incorrect vPars software installed on the boot device.
Error loading "ANALYTICS". Shutting down "ANALYTICS"...
[MON] ANALYTICS has halted.

MON>

Re: System won't boot after kernel parameter change

Don Morris_1 — Tue, 03 Feb 2009 16:40:05 GMT

Ah... ok.. so it swapped last_install into the current configuration, but last_install was installed as an nPar not a vPar (and there are missing kernel modules needed).

Do you still have the ls output of the available subdirectories under /stand? All we need to do is find one of your vPar configurations -- preferably not the one you were running, but if we must then you can get back to booting that in failsafe mode [one of the things Failsafe would have done is explicitly copy that configuration to a unique name it displayed on the console... so that must be there].

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 17:00:42 GMT

Ok. We booted from /stand/backup/vmunix and entered failsafe mode. We'll try to reset all kernel parameters to default with kctune.

Owen

Re: System won't boot after kernel parameter change

Kuekuatsheu — Tue, 03 Feb 2009 17:13:35 GMT

Tried kctune and reset all values. Still won't boot. Can I recreate my /stand/vmunix using my backup kernel in /stand/backup/vmunix? Just to try and see if the system will boot.

Owen

Re: System won't boot after kernel parameter change

Dennis Handly — Tue, 03 Feb 2009 17:27:04 GMT

>I changed the kernel parameter maxfiles and maxfiles_lim to 409600.

This looks like the results of your other thread:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1310293

400K is way too big. Who needs that many open files/sockets?

Re: System won't boot after kernel parameter change

Avinash20 — Tue, 03 Feb 2009 18:08:08 GMT

If you are able to boot from /stand/backup/vmunix, then just copy /stand/backup/vmunix to /stand/vmunix and boot the system