HP-UX server crash

[John Wesley]

Hi:

My name is John Wesley. Few days back, we experienced an over-night intermittent shutdown of our power generator machine, which ended up crashing our HP-UX server. This server I'd say is pretty old though: HP 9000 D-Class. Now, not all disks are mounted during startup. Have physically shutdown, removed and put back the disks, yet no solution. Will appreciate if someone could please kindly help me work through for an urgent fix as there critical applications that needs to be accessed on that server.

[RAC_1]

Are you saying that system is not coming up? Or system is coming up, but not all FS are mounted?

[John Wesley]

Hi RAC,

That's right, not all FS are been mounted or coming up.

[RAC_1]

Take server into single user mdoe ,and comment out all mounts except vg00. Reboot. Does it come up?

[John Wesley]

RAC,

After a reboot, the situation remains the same. Only vg00 is mounted. vg02 and vg03 are still not available.

[Bill Hassell]

Are there any error messages in:

/var/adm/syslog/syslog.log
/etc/rc.log

What happens when you manually activate VG02 and VG03 (ie, vgchange -a y vg02)? If you see error messages about missing or inaccessible disks, you have likely lost some disks and will need to replace them and restore from your latest backup.

[John Wesley]

Hi Bill,

Yes, got error messages in the log files. Extracted few major lines from /etc/rc.log confirming that vg01, vg02 and vg03 are not mounted:

mountall: cannot mount /dev/vg03/lvol32
mountall: diagnostics from mount
mount: Cannot open /dev/vg03/lvol32: no such device or address

The same error message for /dev/vg02/lvol21 and /dev/vg01/lvol11, 13 and 12

"/sbin/rc1.d/S110mbuild start" SKIPPED

[Robert-Jan Goossens]

John,

could you check if the special device files for these filesystems still exists ?

# mount /usr (if not already mounted)

# ll /dev/vg01/ | grep "*lvol1*"

Do you have a good backup of these volume groups?

Robert-Jan

[John Wesley]

Hi Robert,

1) mount /usr
Mount: /dev/vg00/lvol7 is already mounted on /usr

2) ll /dev/vg01/ | grep "*lvol1*"
No feedback

Yes, have a previous backup before the crash. Please advice further.

[Bill Hassell]

The error messages (no such device) are normal if the volume group is not activated. Type in these commands:

vgchange -a y vg01
vgchange -a y vg02
vgchange -a y vg03

If you get I/O errors or errors about the disks not being accessible, This means the disks are dead. If all the disks are in a separate cabinet (not built-in to the D-class), the cabinet may be defective.

[Brian A. Scurlock_1]

An unscientific method:

Upon restart, interupt the boot process. Type "sea all IPL."

This will cause the system to query each disk to see if it is bootable. At some point you should see activity on each disk. If you don't it's a good indication that the ones that fail to show activity during this process are dead.

[Brian A. Scurlock_1]

Better yet, I believe the internal D Class disks are hot plug. Interrupt the bot process and reseat each disk. Should light on re-insertion.
-Brian

[John Wesley]

Did as instructed, yet, the missing VGs are not responding, maybe dead. Shutdown and power-off the entire system; unplug and plu back individual disk; upon startup, all disk were on (power indicator in each), but couldn't continue during the rest of the startup process. Is there no way that these disks be revived?

[IT_2007]

when did you shutdown system last time cleanly?
What stage you are having problem, are you able to boot up into single user mode?

It seems you have hardware problem. Check iosan for any UNCLIAMED or NO_HW disks.

May end up calling HP to fix hardware issues. If you have good backups then create VG's manually along with filesystems and mount points and then restore from tape.

[Bill Hassell]

I'm going to make an assumption that the needed disks are reporting errors in syslog (it's important to look there for disk errors). I'm also going to assume that all the disks are inside the D-class computer (no external box with more disks). Since the system boots up, the controller that talks to the internal disks is probably OK, although without another working D-class, it is not possible to accurately determine if the disks are dead.

I'm also going to assume that none of these disks are mirrored since there would not be enough internal disks to be mirrored with 4 volume groups (I think you indicated that vg00 vg01 vg02 vg03 are part of the same system. So the only choice is to get replacement disks, then use vgcfgrestore to setup each disk, followed by newfs for each logical volume in each volume group, mount the empty lvols and then restore them from your latest backup tape.

If you are without any backups, you can contact some disk recovery companies such as OnTrack.com. NOTE: not many disk recovery companies can handle the high-voltage fastwide SCSI disks used in the D-class, and even some of thos may not be able to handle HP-UX disks. Warning: these services are expensive -- as much as 3 to 10 times the cost of a new disk, but they have good track records in recovering data. It's possible that all the data is intact but the electronics in the disk have been scrambled.

[florin_10]

Hi John,

Can you boot in single user mode? If yes
try following:
a) strings /etc/lvmtab
to determine which disk belongs to what vg
b) ioscan -Cfn disk
to determine if the disks identified above are CLAIMED (seen by the system) or not.
c) iocan -fn
to check if the controller is CLAIMED or not.
d) Contact HP support and provide them with relevant info
Cheers
Flg

[John Wesley]

Dear All,

Sorry for the late response due to time difference. I shall implement your advices shorty and revert to you with outcome. Many thanks.

Regards,
JW