Systems hangs when mirrored boot disk is removed

[Rudy Williams]

I have an rp5430 that has MirrorDisk/UX installed and configured. I mirrored the boot disk and tested a boot from the alternate path. Works just great.

Today I wanted to test how the system will react when the primary boot disk fails. Since they are hot-swap disks, I pulled it out.

While the system remained running, functionality was limited:

- a 'tail -f syslog.log' returned nothing

- a telnet allowed me to enter in my username and password, but would not give me a shell prompt--I know that an entry is written to utmp before the shell is given, so I suspect that the OS could not write to the log

Functionality returned about 20 seconds after I replaced the disk.

I thought that MirrorDisk was supposed to write to the mirror if the first copy is not available.

[harry d brown jr]

Did you mirror ALL LV's for vg00??

Especially swap??

If you did, then can you provide a vgdisplay -v ?


live free or die
harry

[Michael Tully]

As well as the 'vgdisplay -v /dev/vg00' output also include the 'lvlnboot -v'

e.g.

# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c1t6d0 (0/0/2/0.6.0) -- Boot Disk
/dev/dsk/c2t6d0 (0/0/2/1.6.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c1t6d0
/dev/dsk/c2t6d0
Root: lvol3 on: /dev/dsk/c1t6d0
/dev/dsk/c2t6d0
Swap: lvol2 on: /dev/dsk/c1t6d0
/dev/dsk/c2t6d0
Dump: lvol2 on: /dev/dsk/c1t6d0, 0

[Sean OB_1]

Make sure that you've mirrored all of the volumes in vg00. Remember that swap is a volume too and needs to be mirrored.

[S.K. Chan]

I had done such simulation on my L2000 (both my primary and mirrored disks are at the front panel which makes it easy to pull either one out) and it works just fine ! At first after the primary disk is pulled, the syste will "kindda" hang for like probably less than a minute or so and then everything should be running as usual. Doing "vgdisplay -v vg00" will show me that the primary disk is "unavailable". I can only think of one thing .. the ODE (Offline Diagnostics).. not sure it would make a difference here but in your mirroring process the final step should include installing ODE on your mirrored disk. You can confirm if you need to do this by running "lifls" on the primary disk and if you see additional lif files apart from ISL,AUTO,HPUX,PAD and LABEL it means it has ODE and you must then install ODE
on your mirrored disk. The steps are ..(assuming c2t2d0 is the mirrored disk)
# ll /usr/sbin/diag/lif/updatediaglif
=> Make sure the file exist.
# lifls -l /dev/rdsk/c2t2d0
=> List content of LIF dir (it should have ISL,AUTO,HPUX,PAD,LABEL).
# mkboot -b /usr/sbin/diag/lif/updatediaglif -p ISL -p AUTO -p HPUX -p PAD -p LABEL /dev/rdsk/c2t2d0
=> Install ODE LIF files while "protecting" the existing LIF files.
=> VERY IMPORTANT..use updatediaglif2 for 11.0 64-bit.
# lifls -l /dev/rdsk/c2t2d0
=> Should now see the additional ODE LIF files.

[Rudy Williams]

I did mirror all the logical volumes, even swap.

The output of 'vgdisplay -v vg00' and 'lvlnboot -v' are attached.

I tried this test again just now:
- tail -f syslog.log
- pull the disk
- attempt to telnet in (no login recieved this time)
- saw SCSI errors in the syslog
- waited over 2 minutes for the login prompt . . . nothing
- put the disk back in

[Rich Wright]

Be sure that you have "-lq" option set in the AUTO file on each vg00 PDEV.
To verify
lifcp /dev/rdsk/c_t_d_:AUTO -
This should display
hpux -lq

If not, then do
mkboot -a "hpux -lq" /dev/rdsk/c_t_d_

Rich

[Rudy Williams]

Folks--

I found the problem. One of the copies for /dev/vg00/lvol3 (mounted on /) was stale. Look at the vgdisplay -v in my previous post.

I ran 'lvsync' on lvol3, confirmed all the copies were syncd and ran the test again. It passed.

Now, what could cause a mirror copy to become stale? Shouldn't LVM detect stale copies and run lvsync or vgsync automatically? It seems to me that I should not have to manually run this.

cjw

[S.K. Chan]

That might be the explanation why you're having the problem .. due to stale extents. The cause for stale extents is bad section on the disk, sometimes (but very rare) I do noticed vgsync or lvsync can get rid of a stale extent status especially in a situation where you couldn't find any IO error from the disk. In that case I think it's probably due to LVM, not bad sector on the disk. In your situation what may have happened is there might be already some extents marked as stale before you pull the disk out. If so, when the same disk is plug back, auto-synchronization will not happen like you would expect. Thus vgsync or lvsync must be run manually. Just like as if you're inserted a new replacement disk ..for example when a new disk is replaced.. the steps would be ..
Take note of the vgchange and vgsync towards the end.
# mkboot -l /dev/rdsk/c2t2d0
# mkboot -a "hpux -lq" /dev/rdsk/c2t2d0
# vgcfgrestore -n vg00 /dev/rdsk/c2t2d0
# vgchange -a y vg00
# vgsync vg00

[Ceesjan van Hattum]

Hi,

In the ftplog you find 'vmunix: SCSI: Async write error.
In http://bizforums.itrc.hp.com/cm/QuestionAnswer/0,,0x0bafa2db8513d6118ff40090279cd0f9,00.html you will see the same error. Solution there was: firmware upgrade.
Brings me to the thought: what about your last patches?
What OS do you have? 10.20 11.00 11.11 ??

Greetings,
Ceesjan