1832973 Members
2597 Online
110048 Solutions
New Discussion

Re: BAD BLOCKS

 
brian_31
Super Advisor

BAD BLOCKS

Good Morning!!!

Environment 11.11
Database folks are complaining of BAD BLOCKS and they refreshed the database. from the OS side i recommended umounting and doing full fsck on the filesystem in question. Now the customer wants to know the root cause of the issue. lvdisplay, syslog dmesg are all clean. Anything else we can check to provide the root cause for the BAD BLOCKS that oracle experienced?

Thanks

Brian
11 REPLIES 11
Peter Godron
Honored Contributor

Re: BAD BLOCKS

Brian,
anything showing with:
diskinfo -v /dev/rdsk/cxtxdx

IT_2007
Honored Contributor

Re: BAD BLOCKS

Do you looking BAD BLOCKS on local disks or SAN disks?

If you looking for Local then you can check using dd command or ask hp and they can give a binary program which checks bad blocks and also it may be fixed.

dd if=/dev/rdsk/cXtYdZ of=/dev/null bs=4096k

If it is SAN disk then talk to the Vendor.
Pupil_1
Trusted Contributor

Re: BAD BLOCKS

issue the following command on the suspect PV and you should be getting an o/p similar to the one below on a good disk !!

# echo 2400?20X|adb /dev/dsk/c2t0d0
2400: 44454645 43543031 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
#

dd on a failed disk will hang for ever!!
There is always something new to learn everyday !!
Steven E. Protter
Exalted Contributor

Re: BAD BLOCKS

Shalom,

Oracle will probably not tolerate OS bad block relocation.

This is an oracle issue and the database may need to be restored from backup.

If you have archive logging turned on there should not be much data loss.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
David Child_1
Honored Contributor

Re: BAD BLOCKS

Brian,

Is this connected to some type of disk array such as an EMC Symmetrix? Most modern arrays will handle bad block relocation at the hardware level. If you have BB relocation set on an LV and the array detects and moves the block there will be problems. Basically LVM moves the block as does the array so now you have a lost block.

The last time I ran into this the DBAs had to do some recovery. For any LV's built on EMC Symmetrix LUNs you should have BB relocation set to NONE (lvchange -r N /dev//).

I'm sure other arrays (Hitachi, etc.) are similar.

David
David Child_1
Honored Contributor

Re: BAD BLOCKS

Sorry, I forgot;

If the condition I just talked about happens you will not find any bad blocks on your LV because the array already handled it. Your data block has been moved however so that block would be empty.

David
brian_31
Super Advisor

Re: BAD BLOCKS

Thanks. is there a command to check the current bad block allocation, so that once i know i can run the lvchange -r to allocate it to none or what is the default?

Thanks

Brian
David Child_1
Honored Contributor

Re: BAD BLOCKS

I'm not sure if I follow what you are asking at this point. The default is 'on'. You will want to set it for the array you are connected to. What array are you using?

For Symmetrix it needs to be set to NONE (lvchange -r N /dev//). It will show up in lvdisplay as "Bad block NONE".

From the lvchange man page:

"-r N Disable bad block relocation and the Bad Block Directory. Upon a media failure, LVM will NOT attempt to relocate the bad block. In addition it will NOT enter the block in the Bad Block Directory. LVM will have no record of the block being bad, and will attempt to access it on future I/O requests."

This way the array can handle any bad blocks and present only good blocks to the host.

David
TwoProc
Honored Contributor

Re: BAD BLOCKS

Brian,

You can have bad blocks at two different levels, at the file system/hard drive level, and also at the Oracle level. Are the Oracle folks complaining of ORACLE BAD BLOCKS? If so, they can verify the files and where the errors are with the "dbv" tool (from Oracle). Syntax is :

dbv BLOCKSIZE=[YOUR SIZE] FILE=[your file]

e.g.
dbv BLOCKSIZE=8192 FILE=user01.dbf

WARNING: THE ABOVE COMMAND IS *DBA FUNCTION* - not for sysadmin consumption!!!

If this shows an error in a file, you can quickly determine if this is a UNIX issue by running a dd from the file to /dev/null:
dd if=user01.dbf of=/dev/null bs=8k

If there is an error, it should give you an error in the same block as the Oracle error. If it is clean from the dd command, it is very likely that this is an ORACLE ONLY problem, and really much to do with you (other than helping to restore if necessary).

Have the dba's clarify how the BAD BLOCK is manifesting itself, and check it out a bit yourself.

John
We are the people our parents warned us about --Jimmy Buffett
sathish kannan
Valued Contributor

Re: BAD BLOCKS

Hello Brian,

I am not sure where your disks are connected to, either SAN or local internal disk?

If it is a SAN disk it is recommended to turn off the bad block allocation on the logical system level. You SAN will manage bad block allocation by default.

There are tools available (I am not sure who will supply either HP or SAN vendor) to check bad block reallocation on the logical volme level.

To check bad blocks on the SAN, you have to contact your storage vendor to check it.

Short and sweet, turn off bad block reallocation on the logical volume level if connected to SAN.

Regards
Sathish
Don't Think too much
Marek Mahut
Advisor

Re: BAD BLOCKS

Hello,
maybe offtopic, but here is my ksh script for checking bad block setting:

#!/usr/bin/ksh
# writed by Marek Mahut,
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY !
#

echo "\nThis script show bad block settings for each LV: "

ktosom=`/usr/bin/whoami`
if [[ $ktosom = "root" ]]; then

getvgs=`ls -l /dev/*/group | awk '{print $10}' | sed 's/group//g'`
for var in $getvgs ; do
num=`ls -l $var | sort -i | awk '{print $10}' | grep "^lvol" | wc -l`
if [ $num -eq 0 ]; then
bla="blabla"
#echo "\n==>> VG $var has not logical volumes."
else
echo "\n==> Logical volumes for VG $var: "
fi
volu=`ls -l $var | awk '{print $10}' | grep "^lvol" | sort -n`
for vat in $volu ; do
getsetting=`/usr/sbin/lvdisplay -v $var$vat | grep -i "bad block" | awk '{print $3}'`
echo "-> Bad block setting for $var$vat is set to \033[0;1;40;1m$getsetting\033[0m";
done
done

else
echo "\n\033[1;37m\033[41m ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! \033[m"
echo "\033[1;37m\033[41m ! ! ! You are $ktosom, you don't have permissions to run this script ! ! \033[m"
echo "\033[1;37m\033[41m ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! \033[m\n"
fi


echo "\nAnd have a nice day! :)"


all suggestions welcome :)

Marek.
"If I ever find out about someone sacrificing quality in order to meet a shipment schedule, I will personally have him fired." -- Dave Packard