topic Re: boot from SAN snapshot in Operating System - HP-UX

boot from SAN snapshot

Dale Renton — Wed, 17 Jan 2007 08:55:41 GMT

Hello all,

We have two identical HP-UX IA64 boxes. The primary server is booting from an EMC CX300 SAN and doing LVM mirroring to another SAN. On the secondary SAN we have created a snapshot using EMC Snapview. Now we are trying to boot the secondary server off this snapshot for development, testing, etc...

We are unable to boot. When we try to get into LVM maintenance mode it comes back with "no primary swap space configured" among other things.

Anybody have any ideas on how to do this?

Thanks,
Dale

vgdisplay of vg00 (root, swap, dump) from primary server

# vgdisplay -v vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 3
Open LV 3
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4059
VGDA 4
PE Size (Mbytes) 32
Total PE 8098
Alloc PE 8098
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 320
Current LE 10
Allocated PE 20
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 121056
Current LE 3783
Allocated PE 7566
Used PV 2

--- Physical volumes ---
PV Name /dev/dsk/c4t0d0s2
PV Status available
Total PE 4049
Free PE 0
Autoswitch On

PV Name /dev/dsk/c9t0d0s2
PV Status available
Total PE 4049
Free PE 0
Autoswitch On

Re: boot from SAN snapshot

Duncan Edmonstone — Thu, 18 Jan 2007 04:36:16 GMT

Dale,

Do you actually get to a prompt in LVM maintenance mode?

HTH

Duncan

Re: boot from SAN snapshot

Nick W — Thu, 18 Jan 2007 05:32:48 GMT

Dale,

My 2-pence/musings:
- Booting happens in the 'boot area' of the disk
- booting happens *before* LVM kicks in
so...
mirroring at the LVM level is good for data but not enough for booting.

The 'mirroring' needs to happen below LVM (say, at the HW level)

Hope this helps
Nick 'dubya'

Re: boot from SAN snapshot

Dale Renton — Thu, 18 Jan 2007 08:57:22 GMT

We never get to a prompt in LVM maintenance mode.

Attached are some screenshots of the boot process.

What about booting the HP-UX install DVD and running some recovery commands?

Thanks

Re: boot from SAN snapshot

Duncan Edmonstone — Thu, 18 Jan 2007 10:41:01 GMT

YOu shouldn't have to go to the DVD... the only time i've ssen this is when swap was still set up to go to local (internal) disks on the source system. Can we see the output of:

swapinfo

and

lvlnboot -v vg00

on the source system?

Cheers

Duncan

Re: boot from SAN snapshot

Dale Renton — Thu, 18 Jan 2007 10:47:03 GMT

Here you go:

# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 2198092 6190516 26% 0 - 1 /dev/vg00/lvol2
localfs 16777216 0 16777216 0% 16777216 0 2 /paging
reserve - 9633708 -9633708

# lvlnboot -v vg00
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c4t0d0s2 (0/2/1/0.1.0.0.0.0.0) -- Boot Disk
/dev/dsk/c9t0d0s2 (0/5/1/0.4.1.0.0.0.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c4t0d0s2
/dev/dsk/c9t0d0s2
Root: lvol3 on: /dev/dsk/c4t0d0s2
/dev/dsk/c9t0d0s2
Swap: lvol2 on: /dev/dsk/c4t0d0s2
/dev/dsk/c9t0d0s2
Dump: lvol2 on: /dev/dsk/c4t0d0s2, 0

Thanks

Re: boot from SAN snapshot

Duncan Edmonstone — Thu, 18 Jan 2007 11:10:32 GMT

That's strange - this all looks fine - can't see why you wouldn't be able to boot from this... All lvm maintenance mode does is use offsets to find the start of the root, boot and swap partitions on the disk - as long as these lvols are at the *start* of the disk and are all *contiguous* everything should be fine.

I can't see the additional swap space causing a problem (after all you have it set at a lower priority), but you *might* want to try temporarily removing it on the source and repeating the split.

HTH

Duncan

Re: boot from SAN snapshot

Duncan Edmonstone — Thu, 18 Jan 2007 11:18:36 GMT

Actualli Don't know if its just a trick of the formatting, but one line in your boot messages worries me:

entry 0 - auto-configured on root device; NOTICE: USB device attached

the we get some other stuff about USB...

Now like I say, it could be just formatting (everything after the ';' might relate to another subsystem starting), but it does look peculiar... do you have some USB device attache dto this server? Try removing and booting again...

HTH

Duncan

Re: boot from SAN snapshot

Dale Renton — Fri, 19 Jan 2007 10:22:52 GMT

I've found this which I will try :

If the kernel parameter 'swapmem_on' is set to 0, the system
will panic during boot and the following message will be
displayed:

panic: swapconf - no primary swap space configured

If swapmem_on is set to 1, which is the default value, the
system will boot in LVM maintenance mode but the following
additional message will be displayed:

WARNING: no swap device configured, so dump cannot be
defaulted to primary swap.

HP recommends that pseudoswap be enabled
by setting the kernel parameter 'swapmem_on' to the default
value of 1 on systems that boot from an LVM disk.

Ours is set to zero right now.

Thanks,
Dale

Re: boot from SAN snapshot

Dale Renton — Tue, 23 Jan 2007 09:21:57 GMT

Ok, by changing the swapmem_on to 1 we are now able to boot into LVM maintenance mode. Our next step is to get into single user mode with boot -is.

We've made the following changes in LVM maintenance :

1) vgexport vg00
2) mkdir /dev/vg00
3) mknod /dev/vg00/group c 64 0x000000
4) insf -e
5) vgimport /dev/vg00 /dev/dsk/c10t0d0s2
6) vgchange -a y -q n /dev/vg00
7) fsck /dev/vg00/lvol1
8) fsck /dev/vg00/lvol3
9) lvlnboot -r /dev/vg00/lvol3
10) reboot -n

Now when we boot into single user mode it comes back with this error :

ioinit: Rebooting the system because /etc/ioconfig and /stand/ioconfig files are inconsistent

Any suggestions on what to try next? One thing to note is that the insf -e command adds 6 devices (c10t0d0s6), even though it should only be 3. Also we get errors with the vgchange and lvlnboot commands because we only have one of the two LVM mirrors available.

Thanks,
Dale

Re: boot from SAN snapshot

A. Clay Stephenson — Tue, 23 Jan 2007 10:05:45 GMT

This dog ain't gonna hunt. It's really the equivalent of what I do each weekend on all of my boxes -- lifeboat disks. These are done by simply doing a raw disk dd from the boot disk(s) to lifeboat disk(s). In order to actually boot, you have to physically move the lifeboat disk(s) to the boot disk slot(s) so that all the expected i/o paths will be correct. You would have far better success by creating an Ignite image and using that to create your test environment.

Re: boot from SAN snapshot

Duncan Edmonstone — Tue, 23 Jan 2007 10:46:23 GMT

Dale,

The ioconfig thing doesn't surprise me - /stand/ioconfig will have been re-generated when you booted, but /etc/ioconfig will not - I wouldn't worry about that one too much, if the 2nd reboot fixes it.

With respect to the mirror problems, I expect you will need to use a 'lvreduce -m 0' followed by vgreduce -f to resolve this. e.g. if the disk which is now showing as having stale extents is c9t0d0s2 then:

lvreduce -m 0 /dev/vg00/lvol1 /dev/dsk/c9t0d0s2

*repeat for all other lvols*

then

vgreduce -f vg00

proably best to do a lvlnboot -R after all this as well:

lvlnboot -R vg00

As for the apparent additional slices on the disks - that one has me confused - what does diskinfo tell you about them?

HTH

Duncan

Re: boot from SAN snapshot

Dale Renton — Thu, 25 Jan 2007 10:05:09 GMT

How can we do an lvreduce -m 0 /dev/vg00/lvol1 if the mirror volumes are no longer there?

lvdisplay -v /dev/vg00/lvol1

shows PV1 as ??? and PV2 as /dev/dsk/c10t0d0s2

Thanks

Re: boot from SAN snapshot

Dale Renton — Thu, 25 Jan 2007 10:13:21 GMT

Got this one:

lvdisplay -k -v /dev/vg00/lvol1

lvreduce -m 0 -k /dev/vg00/lvol1 0

Re: boot from SAN snapshot

Dale Renton — Mon, 29 Jan 2007 11:28:55 GMT

Here are the finalized steps we've done to complete our boot. For some reason the insf command adds 6 devices when in lvm maintenance, however in single user mode it adds the correct number of 3. Also, we had an stm process consuming %50 cpu that we killed. (maybe due to the fact that the server's network connection is unplugged ?)

1) boot -lm
2) vgexport vg00
3) mkdir /dev/vg00
4) mknod /dev/vg00/group c 64 0x000000
5) insf -e -H
6) vgimport /dev/vg00 /dev/dsk/c0t3d0s2
7) vgchange -a y -q n /dev/vg00
8) lvreduce -m 0 -k /dev/vg00/lvol1 0
9) lvreduce -m 0 -k /dev/vg00/lvol2 0
10) lvreduce -m 0 -k /dev/vg00/lvol3 0
11) vgreduce -f vg00
12) /sbin/fs/vxfs/fsck /dev/vg00/lvol1
13) /sbin/fs/vxfs/fsck /dev/vg00/lvol3
14) ioinitrc
15) boot -is
16) rmsf -H
17) insf -e -H
18) init 3

Thanks all for your help.