System with 12H wont't boot: vxfs sanity check

[Maarten van Maanen]

We have a K370 with a 12H Autoraid. When changing volumes on the 12H the system hung on mounting a changed volume. No login was possible after that.
I hard rebooted the system by cycling the power. On initializing the 12H reported Disk B2 failed. The system tries to boot and gives POWERFAILED errors, log replay in progress, vxfs: sanity check.
The 12H shows activity on the controllers. System console says it's checking vxfs file systems but it seems to take forever.
My question is: Should I wait or reboot the system without the 12H.

[Sridhar Bhaskarla]

Hi Maarten,

Looks like the volume groups might have been corrupted. Boot into single user mode and try to activate each volume group and check for
errors.

#strings /etc/lvmtab |grep vg
#vgchange -a y vg??
#fscck /dev/vg??/lvol?
#mount /dev/vg??/lvol? /mount

And also look at the /var/adm/syslog/syslog.log for any errors.

-Sri

[A. Clay Stephenson]

Hi Maarten:

It is not clear what you mean by 'changing volumes'. Did you remove a LUN and the host knows nothing about it? Srihar's reply will probably get you started but it would help to know what 'changing volumes' mean lest we accidently lead you astray. Your 12H should have no difficulty from recovering from one failed disk.

Regards, Clay

[Maarten van Maanen]

How do I reboot in singel user mode if the system is still busy doing it's vxfs: sanity check.
Message on console are:

log replay in progress
replay complete - marking super-block as CLEAN:
/dev/vg01/lvol1:
LVM: vg1[1]: pvnum=0 (dev_t=0x1f001001000) is POWERFAILED

Our 12H seems to be very busy judging by the controllers but otherwise there is no activity. Has been like this for two hours know. I don't no if I should wait or just try another reboot.

[Maarten van Maanen]

I had to change two volumes. One I was reducing from 6 to 1Gb, the other had to be increased from 5 to 10. This way I didn't need any new LUN's and could keep my unused space for RAID 0/1.
I think I made a mistake by not creating a new fs on the reduced volume and then tried to mount the lvol. On that the complete system got hung up. Locked my console and refused any new connections.

[A. Clay Stephenson]

Yep, I think you clobbered yourself. There is no really good answer here but I would power off the server and boot to single-user mode. I would then mount /var and /usr so that you have vi and edit /etc/fstab so that these two file systems do not mount. I would then reboot
and rebuild/resore the lvols and filesystems after the 12H finishes its light show. I hope that you have good backups.

[Maarten van Maanen]

Ok, can reboot the server
How do I reboot in single user mount after I've powered down the system.
And I quess I should not power down the 12H as well when I do reboot the systeem?

[A. Clay Stephenson]

Hi again Maarten:

I would definitely leave the 12H on until it finishes its rebuild (which can take quite a while). I would also contact support and get a replacement drive. Again, when you remove the bad disk and insert the new disk, be patient because it make take 3 or 4 hours to balance.

[Maarten van Maanen]

Hi,

Thanks for your response. have rebooted the system with hupx -si and changed the fstab entries. However, system will still not boot because of the 12H. BTW, the 12H is not doing anything. I've got two amber LED's on two disks while the controllers seem to be very busy. According to the control panel a rebuild is "suggested". Array won't shutdown anyway. Seems I'm stuck with this until I get at least one replacement drive from HP.

[Bill McNAMARA_1]

you can't reduce a lun without deleting and recreating the lun.
ie: there is no reduce lun option and pulling a disk is not lun reduce..