topic Some error messages abt the storage in Operating System - Linux

Some error messages abt the storage

Waqar Razi — Wed, 05 Aug 2009 13:32:45 GMT

We have a RHEL server and we are getting the following error messages in the syslog:

Aug 2 04:07:04 chw-hpm-001 multipathd: sdfa: directio checker reports path is d
own
Aug 2 04:07:04 chw-hpm-001 kernel: sd 1:0:1:8: SCSI error: return code = 0x0001
0000
Aug 2 04:07:04 chw-hpm-001 multipathd: sdfb: directio checker reports path is d
own
Aug 2 04:07:04 chw-hpm-001 kernel: end_request: I/O error, dev sden, sector 0
Aug 2 04:07:04 chw-hpm-001 multipathd: sdfc: directio checker reports path is d
own
Aug 2 04:07:04 chw-hpm-001 kernel: sd 1:0:1:9: SCSI error: return code = 0x0001
0000
Aug 2 04:07:04 chw-hpm-001 multipathd: sdfd: directio checker reports path is d
own
Aug 2 04:07:04 chw-hpm-001 kernel: end_request: I/O error, dev sdeo, sector 0
Aug 2 04:07:04 chw-hpm-001 multipathd: sdfe: directio checker reports path is d
own

Can some one please give me some clue.

Re: Some error messages abt the storage

Steven E. Protter — Wed, 05 Aug 2009 14:20:51 GMT

Shalom,

At one point the system detected storage at the location given. Then it was removed, or was detected.

Linux keeps an annoying record of this.

/dev/sda
Then
/dev/sdb

The messages themselves can be ignored, or you can try removing the device files in /dev

Your choice.

SEP

Re: Some error messages abt the storage

Waqar Razi — Wed, 05 Aug 2009 14:33:19 GMT

Can you please explain it further. We have a RAC cluster and we are having the same messages on all the servers included in the cluster.

Re: Some error messages abt the storage

Steven E. Protter — Wed, 05 Aug 2009 15:44:03 GMT

Shalom,

Linux tries to keep a record of disk discovery. This is particular to my experience on RHEL.

The first disk discovered is /dev/sda
Then /dev/sdb
/dev/sdc

and so on.

So lets say I have a four disk server, and I build it with two hardware raid 1 disks presented.

I do an OS install. RHEL 5, update 3 for example.

/dev/sda is the boot disk.
/dev/sdb is the second disk intended for data.

Everything is hunky dory, then /dev/sdb gets full.

I add two fiber channel hba cards to the system and connect them both the the SAN.

I present a gigantic LUN to the system.

It becomes /dev/sdc.

Then I migrate the data from /dev/sdb to /dev/sdc

Times being tough, I want to use the two disks that comprised /dev/sdb to boot another system. I boot the server, remove them from raid coniguration and physically remove them from the server.

I reboot.

Now I am getting a bunch of errors like yours I/O on /dev/sdb

Its not there any more but the server kept a record of it.

So I get the errors. Do I care? Not really, I migrated the data.

Now remember I put in two hba fiber cards.

Well Linux has built in multi pathing, so not only do I have /dev/sdc but I have /dev/sdd

Default multipath configuration is active passive, so the system starts getting the same I/O errors on /dev/sdd because the path is inactive.

Now you know the cause.

Lets see about fixing it.

One step would be to go to full multi path active active coniguration. So here is a link on how to do that.

http://www.redhat.com/docs/manuals/csgfs/browse/4.6/DM_Multipath/index.html

http://www.tuxyturvy.com/blog/index.php?/archives/42-RHEL4-dm-multipath-on-root-HOWTO.html

That solves the /dev/sdd problem.

But what about /dev/sdb

We still get that annoying error message because we pulled the disk.

Now I find this to be inconsequential, so I just ignore it. Then I have an anal moment and I decide to fix it.

The first thing I'd try and do is use fdisk to remove all configuration information on the disk.

fdisk /dev/sdb

This is a start.

I'm going to do a little more research on this and try and get you a better idea as your post reminds me this problem annoyed the hell out of me.

SEP

Re: Some error messages abt the storage

Steven E. Protter — Wed, 05 Aug 2009 15:50:22 GMT

If this issue is caused by multi-path, properly configuring multipath will fix it.

Try
sfdisk -R /dev/sdfc
sfdisk -R /def/sdfe

Make sure you have a backup before you start.

You don't want to dink the data.

SEP