CANNOT READ : BLK 232

[Henry Chua]

Hi

I have a problem starting my HPUX server after a power outage. I was prompted the following during the initial boot up. And thereafter when fsck was invoked :
….
/bin/bcheckrc:
Checking for LVM volume groups and activating (if any exist)
Volume Group “/dev/vg09” has been successfully changed
vxfs fsck: sanity check: root filesystem ok (mounted read/write)
checking hfs file systems
/bin/fsclean : /dev/vg09/stand (mount) ok
/bin/fsclean: /dev/vg09/lvcustomer not clean, run fsck

“FILE SYSTEM NOT PROPERLY SHUTDOWN, BEGINNING FILE SYSTEM REPAIR”

/dev/vg09/lvcustomer: CANNOT READ : BLK 232
/dev/vg09/lvcustomer: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

/dev/vg09/rstand: 21files, 0 iconf, 22161used, 25668 free L12 frags; 3207 blocks

Could not fix system with fsck -p, RUN INTERACTIVELY

Logging in as root FOR MANUAL fsck. Enter ^D when file system fixed

(bcheckrc)# fsck

…
CANNOT READ: BLK 232
CONTINUE? Yes

…
vxfs fsck :/dev/vg09/root
cannot open device /dev/vg09/usr
cannot open device /dev/vg09/opt
vxfs fsck: cannot read OLT extend1
read of OLT copy failed
vxfs fsck: cannot initialize aggregate
cannot open device /dev/vg09/var

fsck –y, prompted me the same error
I tried “fsck –F –o full /dev/vg09/var” but it prompted that the file cannot be opened.
Also tried force mounting “mount –o ro /dev/vg09/var /mnt”, didn’t help
I tried to use “dd” command to see whether there is any hardware error on my drives but the system prompted me an invalid command comment.

What could be the problem and how can I resolve this?

I thank you all for your suggestion.

[G. Vrijhoeven]

Hi,

Did you try a :
fsck -F vxfs -o full /dev/vg09/rvar ?

Gideon

[Olivier Decorse]

Hi,
1) First, try to mount /usr to access to all unix commands, with :
mount /usr
2) chech your disk with :
/usr/bin/dd if=/dev/dsk/cxtxd0 of=/dev/null bs=1024
and look for any error,
3) OLT stands for "Object Location Table" & is very important to find the inode of the file. It's a critical part of the metadata filesystem area !
4) i heard that fsck_vxfs does not search for all possible alternate super blocks, and that it is a know bug (document JAGaa53168).
To get a fix, please contact your local HP Support Center.
5) if no fix exists, you will need to recreate the FS and restore any data from backup.

Olivier.

[Steve Lewis]

Options:

1.You could try the dd from the raw disk device like this:
dd if=/dev/rdsk/c1t2d0 of=/dev/null bs=2048k

2. If that fails at some point and the full fsck didn't work, then the disk is likely to be toast. Replace the disk and get out your make_recovery tapes.

3. Try (in single user mode), creating a new /var filesystem in the logical volume and running /sbin/frecover from your last fbackup tape.

4. If mirrored, then you could try removing one disk and checking the other one.

[Henry Chua]

Hi Gideon,
The command prompted
"cannot open device /dev/vg09/rvar"

Hi Oliver,
The "mount /usr" command prompted
"vxfs mount : I/O error"
"vxfs mount : cannot mount /dev/vg09/usr"

Thank you guys for your help... is there anyway I can bypass blk 232 and bootup the system?

Many thanks to ALL!!

[Olivier Decorse]

can you :

1) type /bin/bdf to see what is already mounted, and
2) type /etc/fstab to see what need to be mount.

Olivier.

[G. Vrijhoeven]

Hi,

Could you do a

# ll /dev/vg09
# vgdisplay -v vg09

if no device files present:
# ioscan -fnCdisk
# insf

Gideon

[Henry Chua]

Hi Steve,

Thanks for your adviseI will try doing that.

Hi Oliver,

Both commands can't really work,
1. yield "/sbin/sh: /bin/bdf: not found"
2. "/sbin/sh: /etc/fstab: cannot execute"

Hi Gideon,

"ll" can't work without "usr" mounting, but there are files in /dev/vg09.. I can view them using "ls". "vgdisplay -v vg09" did show a list of files..

Anyhow I tried "ioscan -fnCdisk" command and also "insf".. the first one shows three device(my system have 2 drives). May I know whats the rational for using these commands, how shall I proceed thereafter?

Thank you guys for your fantastic help!

[Bill Hassell]

You have a hardware failure, specifically a bad spot on the disk. fsck reports that in trying to fix the directory structures, it was unable read required data on the disk. fsck does not fix anything in the data area, just with the directory structures. Because there is a bad spot in the structures, fsck (and HP-UX) cannot make any sense out of the filesystem. You should also see lots of errors in /var/adm/syslog/syslog.log. There are only a few choices:

1. replace the bad disk and restore your system from your Ignite/UX backup tape

2. hire an HP-UX filesystem expert to manually go through the disk and patch around the bad spot(s) on the disk. This could take several days...

3. Send the disk to a disk recovery company that understands HP-UX systems. This will cost several thousand dollars but may be your only choice if step #1 is not possible.

[Olivier Decorse]

ok Henry,
lets start à the beginning because I do some syntax mistakes and have to correct it :

1) what is your server model ? Only to know if your disks are hot-plug, and can be extract. Do it, just to verify any connection problem.
2) As you said, you see 3 lines at the ioscan -fnkC disk. The last is certainely for the CD/DVD. Is the S/W State : CLAIMED, NO_HW, or ERROR, for the 2 disks ?
3) To verify what is mounted, do (and let us know) :
/sbin/mount
4) Try repairing /usr with :
/sbin/fsck -F vxfs -o full /dev/vg09/usr
5) Hoping last command works, you can now mount /usr with :
/sbin/mount /usr
6) and finaly check your 2 disks (replace cxtydz with the result of the ioscan), with :
/usr/bin/dd if=/dev/rdsk/cxtxd0 of=/dev/null bs=1024

Olivie

[Henry Chua]

Hi Bill,

Thank you for your advise, regarding your first suggestion, unfortunately there wasn't any backup done - the system has been shutdowned for a period of time prior to this. Anyway how does the Ignite software works.. How can i start backing up my other system effectively??

Hi Olivier..

Unfortunately, the system reside in a very old equipment. I do not have any document on it, is there anyway I can find out what model is it?
ok, the ioscan -fnkC disk produce "as per attached" and the /sbin/mount yield:
" /on/dev/vg09/not log on mon sep 6 08:31:04 2004..
/stand on /dev/vg09/stand default on mon sep 6 08:31:06 2004.."
the fsck repair u suggested didn't work... system prompted I/O error can execute.

.. IF I Need to install a new harddisk how should I go about it?

Thank you all for your invaluable advises!

[Bill Hassell · Accepted solution]

First, you'll need to determine the computer model and version of HP-UX. Type the commands:

model
uname -a

As far as Ignite/UX, this is a boot-disk recovery tool and can be downloaded for free from http://www.software.hp.com/products/IUX/overview.html This asumes that you are at least running version 10.xx or greater. Note that all versions of HP-UX 10. (and earlier) are obsolete and unsupported. You'll see the warnings in the Ignite/UX web pages.

As far as a backup policy, you need to run the makle_tape_recovery program from Ignite/UX every few weeks (you'll need a tape drive) and run fbackup at least weekly if not daily. SAM can schedule a full backup for you.

As far as replacing the disk, there are several dozen models from small workstations to giant servers, each with very different hardware so we'll need the model information first. From the ioscan for disks, it looks like this is a workstation.

[Henry Chua]

THANK YOU ALL FOR UR SUPPRT!!

[Henry Chua]

it appears 60% solved.. another 40% up to myself.. may the force be with me!