topic Re: BAD BLOCKS in Operating System - HP-UX

BAD BLOCKS

brian_31 — Tue, 22 Aug 2006 07:56:00 GMT

Good Morning!!!

Environment 11.11
Database folks are complaining of BAD BLOCKS and they refreshed the database. from the OS side i recommended umounting and doing full fsck on the filesystem in question. Now the customer wants to know the root cause of the issue. lvdisplay, syslog dmesg are all clean. Anything else we can check to provide the root cause for the BAD BLOCKS that oracle experienced?

Thanks

Brian

Re: BAD BLOCKS

Peter Godron — Tue, 22 Aug 2006 08:03:53 GMT

Brian,
anything showing with:
diskinfo -v /dev/rdsk/cxtxdx

Re: BAD BLOCKS

IT_2007 — Tue, 22 Aug 2006 08:13:06 GMT

Do you looking BAD BLOCKS on local disks or SAN disks?

If you looking for Local then you can check using dd command or ask hp and they can give a binary program which checks bad blocks and also it may be fixed.

dd if=/dev/rdsk/cXtYdZ of=/dev/null bs=4096k

If it is SAN disk then talk to the Vendor.

Re: BAD BLOCKS

Pupil_1 — Tue, 22 Aug 2006 08:27:00 GMT

issue the following command on the suspect PV and you should be getting an o/p similar to the one below on a good disk !!

# echo 2400?20X|adb /dev/dsk/c2t0d0
2400: 44454645 43543031 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
#

dd on a failed disk will hang for ever!!

Re: BAD BLOCKS

Steven E. Protter — Tue, 22 Aug 2006 08:47:56 GMT

Shalom,

Oracle will probably not tolerate OS bad block relocation.

This is an oracle issue and the database may need to be restored from backup.

If you have archive logging turned on there should not be much data loss.

SEP

Re: BAD BLOCKS

David Child_1 — Tue, 22 Aug 2006 08:57:05 GMT

Brian,

Is this connected to some type of disk array such as an EMC Symmetrix? Most modern arrays will handle bad block relocation at the hardware level. If you have BB relocation set on an LV and the array detects and moves the block there will be problems. Basically LVM moves the block as does the array so now you have a lost block.

The last time I ran into this the DBAs had to do some recovery. For any LV's built on EMC Symmetrix LUNs you should have BB relocation set to NONE (lvchange -r N /dev//).

I'm sure other arrays (Hitachi, etc.) are similar.

David

Re: BAD BLOCKS

David Child_1 — Tue, 22 Aug 2006 08:58:44 GMT

Sorry, I forgot;

If the condition I just talked about happens you will not find any bad blocks on your LV because the array already handled it. Your data block has been moved however so that block would be empty.

David

Re: BAD BLOCKS

brian_31 — Tue, 22 Aug 2006 09:22:20 GMT

Thanks. is there a command to check the current bad block allocation, so that once i know i can run the lvchange -r to allocate it to none or what is the default?

Thanks

Brian

Re: BAD BLOCKS

David Child_1 — Tue, 22 Aug 2006 14:17:42 GMT

I'm not sure if I follow what you are asking at this point. The default is 'on'. You will want to set it for the array you are connected to. What array are you using?

For Symmetrix it needs to be set to NONE (lvchange -r N /dev//). It will show up in lvdisplay as "Bad block NONE".

From the lvchange man page:

"-r N Disable bad block relocation and the Bad Block Directory. Upon a media failure, LVM will NOT attempt to relocate the bad block. In addition it will NOT enter the block in the Bad Block Directory. LVM will have no record of the block being bad, and will attempt to access it on future I/O requests."

This way the array can handle any bad blocks and present only good blocks to the host.

David

Re: BAD BLOCKS

TwoProc — Tue, 22 Aug 2006 14:27:32 GMT

Brian,

You can have bad blocks at two different levels, at the file system/hard drive level, and also at the Oracle level. Are the Oracle folks complaining of ORACLE BAD BLOCKS? If so, they can verify the files and where the errors are with the "dbv" tool (from Oracle). Syntax is :

dbv BLOCKSIZE=[YOUR SIZE] FILE=[your file]

e.g.
dbv BLOCKSIZE=8192 FILE=user01.dbf

WARNING: THE ABOVE COMMAND IS *DBA FUNCTION* - not for sysadmin consumption!!!

If this shows an error in a file, you can quickly determine if this is a UNIX issue by running a dd from the file to /dev/null:
dd if=user01.dbf of=/dev/null bs=8k

If there is an error, it should give you an error in the same block as the Oracle error. If it is clean from the dd command, it is very likely that this is an ORACLE ONLY problem, and really much to do with you (other than helping to restore if necessary).

Have the dba's clarify how the BAD BLOCK is manifesting itself, and check it out a bit yourself.

John

Re: BAD BLOCKS

sathish kannan — Wed, 23 Aug 2006 03:33:21 GMT

Hello Brian,

I am not sure where your disks are connected to, either SAN or local internal disk?

If it is a SAN disk it is recommended to turn off the bad block allocation on the logical system level. You SAN will manage bad block allocation by default.

There are tools available (I am not sure who will supply either HP or SAN vendor) to check bad block reallocation on the logical volme level.

To check bad blocks on the SAN, you have to contact your storage vendor to check it.

Short and sweet, turn off bad block reallocation on the logical volume level if connected to SAN.

Regards
Sathish

Re: BAD BLOCKS

Marek Mahut — Wed, 23 Aug 2006 03:39:23 GMT

Hello,
maybe offtopic, but here is my ksh script for checking bad block setting:

#!/usr/bin/ksh
# writed by Marek Mahut,
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY !
#

echo "\nThis script show bad block settings for each LV: "

ktosom=`/usr/bin/whoami`
if [[ $ktosom = "root" ]]; then

getvgs=`ls -l /dev/*/group | awk '{print $10}' | sed 's/group//g'`
for var in $getvgs ; do
num=`ls -l $var | sort -i | awk '{print $10}' | grep "^lvol" | wc -l`
if [ $num -eq 0 ]; then
bla="blabla"
#echo "\n==>> VG $var has not logical volumes."
else
echo "\n==> Logical volumes for VG $var: "
fi
volu=`ls -l $var | awk '{print $10}' | grep "^lvol" | sort -n`
for vat in $volu ; do
getsetting=`/usr/sbin/lvdisplay -v $var$vat | grep -i "bad block" | awk '{print $3}'`
echo "-> Bad block setting for $var$vat is set to \033[0;1;40;1m$getsetting\033[0m";
done
done

else
echo "\n\033[1;37m\033[41m ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! \033[m"
echo "\033[1;37m\033[41m ! ! ! You are $ktosom, you don't have permissions to run this script ! ! \033[m"
echo "\033[1;37m\033[41m ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! \033[m\n"
fi

echo "\nAnd have a nice day! :)"

all suggestions welcome :)

Marek.