Server would not boot after replacing mirror in vg00

[Scott Hillman]

Second disk, the mirror disk, in the vg00 volume group has stale partitions that won't synchronize.

NSGBU1:/root# lvsync /dev/vg00/lvol1
lvsync: Couldn't re-synchronize stale partitions of the logical volume:
Device offline/Powerfailed

I get this for all LV that I attempt to synchronize. When HP CE replaced the disk, the server would not boot up. He had to put the problem disk back in the server. This was my final indicatin that the disk is bad.

NSGBU1:/root# dd if=/dev/rdsk/c0t10d0 of=/dev/null bs=1024k
dd read error: I/O error
8+0 records in
8+0 records out

Looks like the primary disk in vg00 should boot based on this command output.

NSGBU1:/root
# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c0t5d0 (8/4.5.0) -- Boot Disk
/dev/dsk/c0t10d0 (8/4.10.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c0t5d0
/dev/dsk/c0t10d0
Root: lvol3 on: /dev/dsk/c0t5d0
/dev/dsk/c0t10d0
Swap: lvol2 on: /dev/dsk/c0t5d0
/dev/dsk/c0t10d0
Dump: lvol2 on: /dev/dsk/c0t5d0, 0

What am I missing?

[Denver Osborn]

What's the output of...

lifls -l /dev/rdsk/c0t5d0
lifls -l /dev/rdsk/c0t10d0

If they show "not a lif volume", then someone forgot to mkboot.

What error did you get when attempting to boot? If it fails again, can you get to lvm main mode? If you can do lvm maint mode, trt "lvlnboot -R" to make sure the BDRA gets updated.

-denver

[Raj D.]

Scott,

Can you boot the system with hpux -lq , (After replacing the disk) and interacting with ISL. If you can then you can remirror the new disk after booting.


Cheers,
Raj.

[Scott Hillman]

This is the output of the lifls commamds.



NSGBU1:/root# lifls -l /dev/rdsk/c0t5d0
volume ISL10 data size 7984 directory size 8 97/11/06 02:10:13
filename type start size implement created
===============================================================
ISL -12800 584 240 0 97/11/06 02:10:13
AUTO -12289 824 1 0 97/11/06 02:10:13
HPUX -12928 832 800 0 97/11/06 02:10:13
PAD -12290 1632 1700 0 97/11/06 02:10:14
LABEL BIN 3336 8 0 01/01/31 09:24:29
SSINFO -12286 3344 2 0 06/12/20 21:31:21
NSGBU1:/root# lifls -l /dev/rdsk/c0t10d0
volume ISL10 data size 7984 directory size 8 97/11/06 02:10:13
filename type start size implement created
===============================================================
ODE -12960 584 704 0 98/07/15 09:53:04
MAPFILE -12277 1288 64 0 98/07/15 09:53:05
SYSLIB -12280 1352 307 0 98/07/15 09:53:05
CONFIGDATA -12278 1664 195 0 98/07/15 09:53:05
SLMOD -12276 1864 109 0 98/07/15 09:53:05
SLDEV -12276 1976 175 0 98/07/15 09:53:05
SLDRIVERS -12276 2152 338 0 98/07/15 09:53:05
SLSCSI -12276 2496 118 0 98/07/15 09:53:05
MAPPER -12279 2616 187 0 98/07/15 09:53:05
IOTEST -12279 2808 203 0 98/07/15 09:53:05
PERFVER -12279 3016 199 0 98/07/15 09:53:06
PVCU -12801 3216 65 0 98/07/15 09:53:06
SSINFO -12286 3288 2 0 98/07/15 09:53:06
ISL -12800 3296 240 0 97/11/06 02:10:13
HPUX -12928 3536 800 0 97/11/06 02:10:13
AUTO -12289 4336 1 0 97/11/06 02:10:13
LABEL BIN 4344 8 0 01/04/11 12:34:27

Also, the server is in a data center, and it didn't have a console attached it, so the only alternative was to have HP put the old drive back in.

[Raj D.]

Also you need to check
# lifcp /dev/rdsk/c0t5d0 :AUTO -

Cheers,
Raj

[Raj D.]

The command is:
# lifcp /dev/rdsk/c0t5d0:AUTO -

no space after the disk device number.

[Scott Hillman]

This is what I get Raj.

NSGBU1:/usr/sbin/diag/lif# lifcp /dev/rdsk/c0t5d0:AUTO -
hpux (;0)/stand/vmunix

Thanks,
Scott

[George Liu_4]

pull out c0t10d0 and try boot again.

[Andrew Rutter]

hi,

your lif isnt set up to boot with no quorum, slight oversight when the mirror was done.

boot the system to pdc and do a fresh search

MA> sea

then boot the boot disk and select interact with ipl

then at ISL prompt boot with quorum option

ISL> hpux -lq

this will boot the mirror.

when you have repalced your bad disk and re-synced everything back up you should then change the lif to include the no quorum option so that in future it will boot up on one disk if needed

Andy

[Michael Steele_2]

To replace a boot disk, ( after booting onto the alternate ),

# vgcfgrestore -n /dev/vg00 /dev/rdsk/cXtXdx

# vgchange -a y /dev/vg00 #

# vgsync /dev/vg00

# mkboot -v /dev/rdsk/cXtXdx

# mkboot -a "hpux -lq" /dev/rdsk/cXtXdx

[Raj D.]

Scott,

Your boot disk has not configured with boot with less qouram option.
So while booting you have to boot it manually with using -lq option.

1. Replace the faulty disk.
2. At main menu :> boot pri
3. Interact with ISL
4. ISL> hpux -lq

5.After system has been booted:
Mirror the new disK:

# vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t10d0

# vgchange -a y /dev/vg00

# vgsync /dev/vg00

(The message will appear : The volume group resycronized, this will take time may be 1hr or more.)

# mkboot -v /dev/rdsk/c0t10d0

# mkboot -a "hpux -lq" /dev/rdsk/c0t10d0


At this point , mirroring to the 2nd disk completed.

6. Reboot and check with normal boot process.



Cheers,
Raj.

[Scott Hillman]

I decided, based on your responses, and the fact that HP could not get back onsite for a few days, that I would remove all of the mirrors and reduce the vg00 for c0t10d0.

Then I followed these steps to recreate the mirror.

pvcreate -B /dev/rdsk/c0t10d0 vgextend /dev/vg00 /dev/dsk/c0t10d0

mkboot /dev/dsk/c0t10d0
mkboot -a "hpux -lq" /dev/rdsk/c0t10d0

mkboot -a "hpux -lq" /dev/rdsk/c0t5d0

lvextend -m 1 /dev/vg00/lvol1
lvextend -m 1 /dev/vg00/lvol2
lvextend -m 1 /dev/vg00/lvol3
lvextend -m 1 /dev/vg00/lvol4
lvextend -m 1 /dev/vg00/lvol5
lvextend -m 1 /dev/vg00/lvol6
lvextend -m 1 /dev/vg00/lvol7
lvextend -m 1 /dev/vg00/lvol8

setboot -a 0/0/2/0.2.0

Now when I look at the LVs, they are all synchd. I can even run the dd and erad the 2nd disk okay.

[Scott Hillman]

As a final confirmation, here's what the LIF on each disk look like.

NSGBU1:/root# lifls -l /dev/rdsk/c0t5d0
volume ISL10 data size 7984 directory size 8 97/11/06 02:10:13
filename type start size implement created
===============================================================
ISL -12800 584 240 0 97/11/06 02:10:13
AUTO -12289 824 1 0 97/11/06 02:10:13
HPUX -12928 832 800 0 97/11/06 02:10:13
PAD -12290 1632 1700 0 97/11/06 02:10:14
LABEL BIN 3336 8 0 01/01/31 09:24:29
SSINFO -12286 3344 2 0 06/12/20 21:31:21

NSGBU1:/root# lifls -l /dev/rdsk/c0t10d0
volume ISL10 data size 7984 directory size 8 97/11/06 02:10:13
filename type start size implement created
===============================================================
ISL -12800 584 240 0 97/11/06 02:10:13
AUTO -12289 824 1 0 97/11/06 02:10:13
HPUX -12928 832 800 0 97/11/06 02:10:13
PAD -12290 1632 1700 0 97/11/06 02:10:14
LABEL BIN 3336 8 0 06/12/22 14:04:17

[Raj D.]

Looks very good, you can also check with lifcp command, to make sure hpux -lq option has been set properly. And check setboot as well.

Cheers,
Raj.

[Scott Hillman]

As mentioned in my comments above, I was able to remove the mirror and recreate it.

Hopefully I can get a chance to test the boot from both disks in the mirror, but that requires a coordinated effort between myself and operations personnel in the data center. Time and resources make this difficult.

Your assistance with this trouble is greatly appreciated.

Thanks,
Scott