HP-UX 10.20 - failing mirrored boot disk (lvm)

[M4rkG]

Hi. I have multiple disks that are reporting as critical (5) on a very old HP9000 server running HP-UX 10.20. This includes the boot disks that are mirrored. I also have non-boot mirrored disks that are failing as well as a failed disk which was not mirrored. HP-UX 10.20 is not my area so I was hoping someone could help advise on the recovery process. The server has no console so booting so single user mode might be a problem. My initial concerns are obviously the boot disk. The primary disk is in a much worse state than the mirror and is showing DD errors. I have managed to source replacement disks. 

Can someone advise how I can replace the primary disk and update it from the mirrored boot disk. Thanks.

[Pradeep82in]

You would find all the information in the below document

HP-UX - When Good Disks Go Bad: Dealing with Disk Failures Under LVM 

[Bill Hassell]

For any server, it is critical to have a working console. Without a console, seeing the hardware status, watching for error messages, reinstalling and getting into single user mode are impossible. You didn't mention what model of computer you are using. That will help with identifying a procedure for to you to follow.

Are both of your boot disks showing errors? If so, replacing one of the disks will require a lot of work. Can the machine be shutdown and then rebooted? Swapping a disk may cause a system panic (crash). I would not try swapping any of the disks until you have a console.

For 10.20 servers, all of them should have a serial console port. You would need an RS232 cable with either a 9pin/25pin or 25pin/25pin cable and a serial terminal. NOTE: there is a 50:50 chance that the cable won't work. When you get the cable, also get a small adapter to swap pins 2 and 3 (commonly called a null modem adapter). Once connected, type ctrl-B on the keyboard to wake up the console connection.

Can you identify which boot disk needs replacing? HINT: Using the dd command can light up the disk activity light. Use the raw device file for input to the dd command. Here is a link that covers HP-UX versions all the way back to version 10:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01911837

[M4rkG]

Hi Bill, I couldn't agree more. Its an old box I've inherited with no supporting data, media, etc. etc.. Its literally sat in a corner waiting to die and now it has everyone is in a panic - its been coming for a long time. That said I'd really like to bring it back to life.  Its an HP9000/D380. This is not my area but as far as I can tell, the primary boot disk is showing errors - I can see this via the dd command. The secondary boot disk is also failing but no dd errors.  I'll try to connect via the RS232 this morning.  Thanks!

Primary Boot Disk:

dd if=/dev/dsk/c0t6d0 of=/dev/null bs=1024k
dd read error: I/O error
91+1 records in
91+1 records out

# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c0t6d0 (8/4.6.0) -- Boot Disk
/dev/dsk/c1t14d0 (8/8.14.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c0t6d0
/dev/dsk/c1t14d0
Root: lvol3 on: /dev/dsk/c0t6d0
/dev/dsk/c1t14d0
Swap: lvol2 on: /dev/dsk/c0t6d0
/dev/dsk/c1t14d0
Dump: lvol2 on: /dev/dsk/c0t6d0, 0

Boots Disk Warnings/Errors:

CRITICAL (5)" for Resource: "/storage/events/disks/default/8_4.6.0" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 62390382 -r /storage/events/disks/default/8_4.6.0 -n 62390277 -a

CRITICAL (5)" for Resource: "/storage/events/disks/default/8_8.14.0" (Thresh old: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 62390406 -r /storage/events/disks/default/8_8.14.0 -n 62390273 -a

[Bill Hassell]

The dd error indicates that there is a bad spot very near the beginning of the disk, so that one is not recoverable.

You said the mirror for root also has errors. Is that based on the diag messages? You can find the hardware path for c1t4d0 with the lssf command:

# lssf /dev/dsk/c1t4d0

The above will provide the hardware path for the mirror. You can also use ioscan like this:

# ioscan -knfC disk

To see the diag message details, copy+paste the commands shown in the message:

/opt/resmon/bin/resdata -R 62390382 -r /storage/events/disks/default/8_4.6.0 -n 62390277 -a

/opt/resmon/bin/resdata -R 62390406 -r /storage/events/disks/default/8_8.14.0 -n 62390273 -a

Note that the D-class computers (your D380) were often sold without the optional console interface. This card plugs into the box and provides the 25 pin serial port. Without that card, there is no console and no way to determine what is happening during a reboot, or to interrupt the reboot in order to start in single user mode.

Unless the errors on the c1t4d0 disk are warnings and not failures, you might be able to replace the c0t6d0 disk while it is running. Once replaced, run ioscan to see if the replacement is working. Then try mirroring from the c1t4d0 disk.

If mirroring fails, you'll need to replace both boot disks and then reinstall with your Ignite tape backup. This will require a working console.

[M4rkG]

Hi Bill, thanks for the help. I managed to connect via an RS232 so I can now boot to single user mode. I tried to boot to the alternative boot disk using "SEA IPL" from the main menu but either I cannot select it or its not detected or the instructions I have are wrong?  Below is what I see.  I dont have an IGNITE tape so by the sound of it I doubt the system is coming back but if you do have any suggestions it would be greatly appreciated. Also if there is any other info you need to help me just let me now.  

SCREEN DUMP:

Main Menu: Enter command > SEA IPL

Searching for device(s) with bootable media...
This may take several minutes.

To discontinue search, press any key (termination may not be immediate).

Path Number  Device Path (dec)    Device Type and Utilities
-----------        ----------------          -------------------------
P0                  8/4.15                      Random access media
                                                     IPL
P1                  8/4.6                        Random access media
                                                     IPL
P2                 8/8.14                       Random access media
                                                     IPL
P3                 8/16/6.0                    LAN Module

Main Menu: Enter command >

[M4rkG]

I'm not sure what this means but it doesn't look good.  The 'stale' entries are from the primary boot disk.  The 'current' entries are from the alternative boot disk.  There are nearly a 1,000 lines.  I have spare disks in the machine that can now be used if required.

# lvdisplay -v /dev/vg00/lvol* | grep stale
LV Status available/stale
0000 /dev/dsk/c0t6d0 0000 stale /dev/dsk/c1t14d0 0000 current
0001 /dev/dsk/c0t6d0 0001 stale /dev/dsk/c1t14d0 0001 current
0002 /dev/dsk/c0t6d0 0002 stale /dev/dsk/c1t14d0 0002 current
0003 /dev/dsk/c0t6d0 0003 stale /dev/dsk/c1t14d0 0003 current
0004 /dev/dsk/c0t6d0 0004 stale /dev/dsk/c1t14d0 0004 current
0005 /dev/dsk/c0t6d0 0005 stale /dev/dsk/c1t14d0 0005 current
0006 /dev/dsk/c0t6d0 0006 stale /dev/dsk/c1t14d0 0006 current
0007 /dev/dsk/c0t6d0 0007 stale /dev/dsk/c1t14d0 0007 current
0008 /dev/dsk/c0t6d0 0008 stale /dev/dsk/c1t14d0 0008 current
0009 /dev/dsk/c0t6d0 0009 stale /dev/dsk/c1t14d0 0009 current
0010 /dev/dsk/c0t6d0 0010 stale /dev/dsk/c1t14d0 0010 current
0011 /dev/dsk/c0t6d0 0011 stale /dev/dsk/c1t14d0 0011 current
0012 /dev/dsk/c0t6d0 0012 stale /dev/dsk/c1t14d0 0012 current
0013 /dev/dsk/c0t6d0 0013 stale /dev/dsk/c1t14d0 0013 current
0014 /dev/dsk/c0t6d0 0014 stale /dev/dsk/c1t14d0 0014 current
0015 /dev/dsk/c0t6d0 0015 stale /dev/dsk/c1t14d0 0015 current
0016 /dev/dsk/c0t6d0 0016 stale /dev/dsk/c1t14d0 0016 current
0017 /dev/dsk/c0t6d0 0017 stale /dev/dsk/c1t14d0 0017 current
0018 /dev/dsk/c0t6d0 0018 stale /dev/dsk/c1t14d0 0018 current
0019 /dev/dsk/c0t6d0 0019 stale /dev/dsk/c1t14d0 0019 current
0020 /dev/dsk/c0t6d0 0020 stale /dev/dsk/c1t14d0 0020 current
LV Status available/stale
0000 /dev/dsk/c0t6d0 0985 stale /dev/dsk/c1t14d0 0985 current
0001 /dev/dsk/c0t6d0 0986 stale /dev/dsk/c1t14d0 0986 current
0002 /dev/dsk/c0t6d0 0987 stale /dev/dsk/c1t14d0 0987 current
0003 /dev/dsk/c0t6d0 0988 stale /dev/dsk/c1t14d0 0988 current
0004 /dev/dsk/c0t6d0 0989 stale /dev/dsk/c1t14d0 0989 current
0005 /dev/dsk/c0t6d0 0990 stale /dev/dsk/c1t14d0 0990 current
0006 /dev/dsk/c0t6d0 0991 stale /dev/dsk/c1t14d0 0991 current
0007 /dev/dsk/c0t6d0 0992 stale /dev/dsk/c1t14d0 0992 current
0008 /dev/dsk/c0t6d0 0993 stale /dev/dsk/c1t14d0 0993 current
0009 /dev/dsk/c0t6d0 0994 stale /dev/dsk/c1t14d0 0994 current
0010 /dev/dsk/c0t6d0 0995 stale /dev/dsk/c1t14d0 0995 current
0011 /dev/dsk/c0t6d0 0996 stale /dev/dsk/c1t14d0 0996 current
0012 /dev/dsk/c0t6d0 0997 stale /dev/dsk/c1t14d0 0997 current
0013 /dev/dsk/c0t6d0 0998 stale /dev/dsk/c1t14d0 0998 current
0014 /dev/dsk/c0t6d0 0999 stale /dev/dsk/c1t14d0 0999 current
0015 /dev/dsk/c0t6d0 1000 stale /dev/dsk/c1t14d0 1000 current
0016 /dev/dsk/c0t6d0 1001 stale /dev/dsk/c1t14d0 1001 current
0017 /dev/dsk/c0t6d0 1002 stale /dev/dsk/c1t14d0 1002 current
0018 /dev/dsk/c0t6d0 1003 stale /dev/dsk/c1t14d0 1003 current
0019 /dev/dsk/c0t6d0 1004 stale /dev/dsk/c1t14d0 1004 current
0020 /dev/dsk/c0t6d0 1005 stale /dev/dsk/c1t14d0 1005 current
0021 /dev/dsk/c0t6d0 1006 stale /dev/dsk/c1t14d0 1006 current
0022 /dev/dsk/c0t6d0 1007 stale /dev/dsk/c1t14d0 1007 current
0023 /dev/dsk/c0t6d0 1008 stale /dev/dsk/c1t14d0 1008 current
0024 /dev/dsk/c0t6d0 1009 stale /dev/dsk/c1t14d0 1009 current
0025 /dev/dsk/c0t6d0 1010 stale /dev/dsk/c1t14d0 1010 current
0026 /dev/dsk/c0t6d0 1011 stale /dev/dsk/c1t14d0 1011 current
0027 /dev/dsk/c0t6d0 1012 stale /dev/dsk/c1t14d0 1012 current
0028 /dev/dsk/c0t6d0 1013 stale /dev/dsk/c1t14d0 1013 current
0029 /dev/dsk/c0t6d0 1014 stale /dev/dsk/c1t14d0 1014 current
0030 /dev/dsk/c0t6d0 1015 stale /dev/dsk/c1t14d0 1015 current
0031 /dev/dsk/c0t6d0 1016 stale /dev/dsk/c1t14d0 1016 current
0032 /dev/dsk/c0t6d0 1017 stale /dev/dsk/c1t14d0 1017 current
0033 /dev/dsk/c0t6d0 1018 stale /dev/dsk/c1t14d0 1018 current
0034 /dev/dsk/c0t6d0 1019 stale /dev/dsk/c1t14d0 1019 current
0035 /dev/dsk/c0t6d0 1020 stale /dev/dsk/c1t14d0 1020 current
0036 /dev/dsk/c0t6d0 1021 stale /dev/dsk/c1t14d0 1021 current
0037 /dev/dsk/c0t6d0 1022 stale /dev/dsk/c1t14d0 1022 current
0038 /dev/dsk/c0t6d0 1023 stale /dev/dsk/c1t14d0 1023 current
0039 /dev/dsk/c0t6d0 1024 stale /dev/dsk/c1t14d0 1024 current
0040 /dev/dsk/c0t6d0 1025 stale /dev/dsk/c1t14d0 1025 current
0041 /dev/dsk/c0t6d0 1026 stale /dev/dsk/c1t14d0 1026 current
0042 /dev/dsk/c0t6d0 1027 stale /dev/dsk/c1t14d0 1027 current
0043 /dev/dsk/c0t6d0 1028 stale /dev/dsk/c1t14d0 1028 current
0044 /dev/dsk/c0t6d0 1029 stale /dev/dsk/c1t14d0 1029 current
0045 /dev/dsk/c0t6d0 1030 stale /dev/dsk/c1t14d0 1030 current
0046 /dev/dsk/c0t6d0 1031 stale /dev/dsk/c1t14d0 1031 current
0047 /dev/dsk/c0t6d0 1032 stale /dev/dsk/c1t14d0 1032 current
0048 /dev/dsk/c0t6d0 1033 stale /dev/dsk/c1t14d0 1033 current
0049 /dev/dsk/c0t6d0 1034 stale /dev/dsk/c1t14d0 1034 current
0050 /dev/dsk/c0t6d0 1035 stale /dev/dsk/c1t14d0 1035 current
0051 /dev/dsk/c0t6d0 1036 stale /dev/dsk/c1t14d0 1036 current
0052 /dev/dsk/c0t6d0 1037 stale /dev/dsk/c1t14d0 1037 current
0053 /dev/dsk/c0t6d0 1038 stale /dev/dsk/c1t14d0 1038 current
0054 /dev/dsk/c0t6d0 1039 stale /dev/dsk/c1t14d0 1039 current
0055 /dev/dsk/c0t6d0 1040 stale /dev/dsk/c1t14d0 1040 current
0056 /dev/dsk/c0t6d0 1041 stale /dev/dsk/c1t14d0 1041 current
0057 /dev/dsk/c0t6d0 1042 stale /dev/dsk/c1t14d0 1042 current
0058 /dev/dsk/c0t6d0 1043 stale /dev/dsk/c1t14d0 1043 current
0059 /dev/dsk/c0t6d0 1044 stale /dev/dsk/c1t14d0 1044 current
0060 /dev/dsk/c0t6d0 1045 stale /dev/dsk/c1t14d0 1045 current
0061 /dev/dsk/c0t6d0 1046 stale /dev/dsk/c1t14d0 1046 current
0062 /dev/dsk/c0t6d0 1047 stale /dev/dsk/c1t14d0 1047 current
0063 /dev/dsk/c0t6d0 1048 stale /dev/dsk/c1t14d0 1048 current
0064 /dev/dsk/c0t6d0 1049 stale /dev/dsk/c1t14d0 1049 current
0065 /dev/dsk/c0t6d0 1050 stale /dev/dsk/c1t14d0 1050 current
0066 /dev/dsk/c0t6d0 1051 stale /dev/dsk/c1t14d0 1051 current
0067 /dev/dsk/c0t6d0 1052 stale /dev/dsk/c1t14d0 1052 current
0068 /dev/dsk/c0t6d0 1053 stale /dev/dsk/c1t14d0 1053 current
0069 /dev/dsk/c0t6d0 1054 stale /dev/dsk/c1t14d0 1054 current
0070 /dev/dsk/c0t6d0 1055 stale /dev/dsk/c1t14d0 1055 current
0071 /dev/dsk/c0t6d0 1056 stale /dev/dsk/c1t14d0 1056 current
0072 /dev/dsk/c0t6d0 1057 stale /dev/dsk/c1t14d0 1057 current
0073 /dev/dsk/c0t6d0 1058 stale /dev/dsk/c1t14d0 1058 current
0074 /dev/dsk/c0t6d0 1059 stale /dev/dsk/c1t14d0 1059 current
0075 /dev/dsk/c0t6d0 1060 stale /dev/dsk/c1t14d0 1060 current
0076 /dev/dsk/c0t6d0 1061 stale /dev/dsk/c1t14d0 1061 current
0077 /dev/dsk/c0t6d0 1062 stale /dev/dsk/c1t14d0 1062 current
0078 /dev/dsk/c0t6d0 1063 stale /dev/dsk/c1t14d0 1063 current
0079 /dev/dsk/c0t6d0 1064 stale /dev/dsk/c1t14d0 1064 current

[Bill Hassell]

It would help to see an ioscan -knfC disk

You can unmirror the bad disk (c0t6d0) using lvreduce -m 0 like this:

# lvreduce -m 0 /dev/vg00/lvol1 /dev/dsk/c0t6d0
# lvreduce -m 0 /dev/vg00/lvol2 /dev/dsk/c0t6d0
# lvreduce -m 0 /dev/vg00/lvol3 /dev/dsk/c0t6d0
# lvreduce -m 0 /dev/vg00/lvol4 /dev/dsk/c0t6d0
etc

When all the vg00 lvols are unmirrored, you can then remove the bad disk:

# lvreduce /dev/vg00 /dev/dsk/c0t6d0

Then find a spare disk and initialize it as a boot disk (pay attention to dsk vs rdsk):

# pvcreate -f -B /dev/rdsk/whatever

Now setup the boot area:

# mkboot /dev/rdsk/whatever
# mkboot -a "hpux -lq" /dev/rdsk/whatever

Now add the new disk into vg00:

# vgextend vg00 /dev/dsk/whatever

Now mirror all the vg00 lvols:

# lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/whatever
# lvextend -m 1 /dev/vg00/lvol2 /dev/dsk/whatever
# lvextend -m 1 /dev/vg00/lvol3 /dev/dsk/whatever
# lvextend -m 1 /dev/vg00/lvol4 /dev/dsk/whatever
etc

Check boot setup with lvlnboot -v

# lvlnboot -v

And check mirroring on all the lvols and the 2 disks in vg00:

# lvlnboot -v
# vgdisplay -v vg00 

Now you should have a stable system with working mirror disks.

[M4rkG]

Thanks for that Bill - that's excellent.  I'll give it a go today!  In the meantime here is the ioscan output. 

# ioscan -knfC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 0 8/4.4.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t4d0 /dev/rdsk/c0t4d0
disk 1 8/4.5.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t5d0 /dev/rdsk/c0t5d0
disk 2 8/4.6.0 sdisk CLAIMED DEVICE IBM DGHS09Y
/dev/dsk/c0t6d0 /dev/rdsk/c0t6d0
disk 13 8/4.13.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t13d0 /dev/rdsk/c0t13d0
disk 6 8/4.14.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t14d0 /dev/rdsk/c0t14d0
disk 7 8/4.15.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
/dev/dsk/c0t15d0 /dev/rdsk/c0t15d0
disk 14 8/8.9.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t9d0 /dev/rdsk/c1t9d0
disk 8 8/8.10.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t10d0 /dev/rdsk/c1t10d0
disk 9 8/8.11.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c1t11d0 /dev/rdsk/c1t11d0
disk 10 8/8.13.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c1t13d0 /dev/rdsk/c1t13d0
disk 11 8/8.14.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t14d0 /dev/rdsk/c1t14d0
disk 12 8/8.15.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t15d0 /dev/rdsk/c1t15d0
disk 5 8/16/5.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0

[Duncan Edmonstone]

oops - minor typo - this step from Bill:

# lvreduce /dev/vg00 /dev/dsk/c0t6d0

Should actually read:

# vgreduce /dev/vg00 /dev/dsk/c0t6d0