topic Re: Determine bad disk from syslog error message. in Operating System - Linux

Determine bad disk from syslog error message.

Steven E. Protter — Wed, 18 Jul 2007 11:37:43 GMT

Believe it or not I've never had to do this before.

I/O error: dev 08:10, sector 0

various sectors.

I need to determine exactly which disk is having problems.

/dev/cciss/c0d0p3 1008M 139M 818M 15% /
/dev/cciss/c0d0p1 463M 20M 420M 5% /boot
/dev/cciss/c0d0p5 5.9G 2.0G 3.6G 35% /home
/dev/cciss/c0d0p7 3.9G 689M 3.0G 18% /opt
none 1005M 0 1005M 0% /dev/shm
/dev/cciss/c0d0p10 2.0G 33M 1.8G 2% /tmp
/dev/cciss/c0d0p8 3.9G 1.5G 2.3G 39% /usr
/dev/cciss/c0d0p6 3.9G 87M 3.6G 3% /var
/dev/cciss/c0d0p9 2.0G 34M 1.8G 2% /var/dtv
/dev/sda5 9.8G 160M 9.1G 2% /oraback/DB2
/dev/sda3 20G 12G 7.4G 61% /oradata/DB1

HP/Compaq Proliant servers.

lspci and dmesg have not been very helpful.

fdisk -l

Disk /dev/sda: 255 heads, 63 sectors, 8855 cylinders
Units = cylinders of 16065 * 512 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 3 24066 83 Linux
/dev/sda2 4 6 24097+ 83 Linux
/dev/sda3 7 2617 20972857+ 83 Linux
/dev/sda4 2618 8855 50106735 5 Extended
/dev/sda5 2618 3923 10490413+ 83 Linux
/dev/sda6 3924 4054 1052226 83 Linux

Unable to read /dev/sdb

Our standard setup involves doing hardware raid with a smart card. Seems we have a dead disk. I need to know which filesystem to run fsck on.

TIA, as always generous points for good answers.

SEP

Re: Determine bad disk from syslog error message.

David Claypool — Wed, 18 Jul 2007 12:16:15 GMT

Since the drive is protected by RAID, most likely your filesystem isn't affected and fsck won't show you anything. You should use the Array Diagnostic Utility.

Re: Determine bad disk from syslog error message.

Stuart Browne — Wed, 18 Jul 2007 15:21:29 GMT

The def 08:10 is your clue. Look for major-8-minor-10 in your /dev (usually /dev/sda10).

Re: Determine bad disk from syslog error message.

Steven E. Protter — Wed, 18 Jul 2007 15:25:30 GMT

Well Stuart, the system does not seem to have a /dev/sda10.

Seems the system is overwhelmed with I/O at the moment can't figure out why it can't read /dev/sdb when its not configured.

Other ideas welcome.

SEP

Re: Determine bad disk from syslog error message.

skt_skt — Wed, 18 Jul 2007 15:25:45 GMT

brw-rw---- 1 root disk 8, 10 Jun 24 2004 sda10

Re: Determine bad disk from syslog error message.

Stuart Browne — Wed, 18 Jul 2007 16:37:46 GMT

Huh.. Weird..

Well, try using fdisk again, but changing the 'units' to sectors instead:

fdisk -lu /dev/sda

There is an 'sdb' in the system? Is it showing up in '/proc/scsi/scsi'?

Re: Determine bad disk from syslog error message.

Court Campbell — Wed, 18 Jul 2007 19:43:47 GMT

/dev/sdb maybe your cdrom.

Re: Determine bad disk from syslog error message.

Court Campbell — Wed, 18 Jul 2007 19:58:07 GMT

what is the output from

# cat /proc/partitions

Re: Determine bad disk from syslog error message.

Steven E. Protter — Thu, 19 Jul 2007 02:15:48 GMT

Shalom

[root@golan1 ~]# ll -u /dev/sdb*
0 brw-rw---- 1 root disk 8, 16 Jul 19 10:15 /dev/sdb
0 brw-rw---- 1 root disk 8, 17 Jul 18 16:32 /dev/sdb1
0 brw-rw---- 1 root disk 8, 21 Jul 18 16:32 /dev/sdb5
0 brw-rw---- 1 root disk 8, 22 Jul 18 16:32 /dev/sdb6

Clearly shows the major and minor number of the problem.

Shared storage is having read errors. If its bad disk on storage, we face downtime. If its just the fiber card or other local system issue, we'll fail over to the working node and correct the hardware problem.

Thannks.

I'll probably close this and hand out points very soon.

SEP

Re: Determine bad disk from syslog error message.

Steven E. Protter — Thu, 19 Jul 2007 02:22:23 GMT

Nice one Court.

I really like /proc/partitions

My output was not from the effected system, it was merely an example to document the case.

Thanks. I should have known that.

SEP

Re: Determine bad disk from syslog error message.

Stuart Browne — Thu, 19 Jul 2007 02:51:56 GMT

Hrm.. You should have covered '/proc/partitions' and 'partprobe' in your RHCE.. Ah well ;)

Re: Determine bad disk from syslog error message.

Steven E. Protter — Thu, 19 Jul 2007 07:39:34 GMT

Partprobe yes use that all the time.

I must have forgotten /proc/partitions

Hopefully not again.

SEP

Re: Determine bad disk from syslog error message.

Stuart Browne — Thu, 19 Jul 2007 15:47:42 GMT

:)

(I just re-did it last week, so it's still fresh)