/dev/sda: read failed after 0 of 4096 at 0: Input/output error

[Danesh Qureshi]

I have installed RHEL AS 4 update 5 on a HP DL740 server. Then I presented 1024MB from the SAN (EVA3000) to the host.

When I type lvmdiskscan I see quite a few read failed input/output errors being reported.

I have attached a file to show you the output of fdisk -l, lssd and lvmdiskscan commands.

How do I fix the above errors?

[Ivan Ferreira]

This could be because:

- The VDISK has READ ONLY mode set.

- You haven't enabled multipath I/O in the HBA driver.

Ensure to have enabled MPIO.

[skt_skt]

are u sure u do not have a disk failure?


Are the VGs getting activated with the errors? Any super block corruption!!

[Danesh Qureshi]

The VDISK is not write protected but enabled.

I am not using secure path software and there are no disk failures on the EVA3000 SAN. All disks are healthy.

All I have done is present 1TGb to the host and then reboot the host.

After rebooting the server, I see lots of device I/O errors in the messages file. I have attached the messages file for you to look at. I have also attached output of the version of the Qlogic firmware and driver version currently installed on the HBA.

Please note I have not even started to use any LVM to create logical volumes and yet these device I/O read errors are being reported soon after I present the 1Tb space to the host.

[skt_skt]

if there was no VG then what are you expecting with the lvmdiskscan which is to list the lvm disks in the system?

[Danesh Qureshi]

I am saying I have not seen this type of behaviour where the system is reporting device I/O errors in the messages file and everytime I issue the pvs or pvscan or any other lvm command then device errors for /dev/sda are being reported.

Under normal circumstances there the system should not been reporting such device I/O errors at all.

Did you have a look at the attached files I provided?

I just wwant to know if these errors can be fixed. Any help would be appreciated.

[Danesh Qureshi]

I am saying I have not seen this type of behaviour where the system is reporting device I/O errors in the messages file and everytime I issue the pvs or pvscan or any other lvm command then device errors for /dev/sda are being reported.

Under normal circumstances the system should not been reporting such device I/O errors at all.

Did you have a look at the attached files I provided?

I just wwant to know if these errors can be fixed. Any help would be appreciated.

[Matti_Kurkela]

I'm not familiar with EVA3000, but I know some SAN storage systems present multiple paths to the disk(s) in an active/passive fashion.

With these storage systems, the host is supposed to primarily use just one path, and switch over to the alternate path if the connection is lost on the primary path. The switchover to the alternate path will take a long time in computer timescale (=lots of milliseconds), so the host needs to be aware of the fact and hold the I/O operations without timing out during the storage path switchover.

If the host attempts to use the alternate path while the primary path is active and processing data, the I/O on alternate path will be rejected - which will cause some error messages.

In this case, you *must* use some multipath-management software (HP SecurePath, Linux device-mapper-multipath) or configure your HBA driver to multipath-aware mode. I'm not sure if all the RHEL 4 AS default HBA drivers support multipath-aware mode: the HBA updated drivers supplied by HP definitely do.

Other SAN storage systems can work in active/active mode: with these, you can access each disk through any and all paths with no special switchovers or other limitations. The multipath-management software or multipath-aware HBA driver is needed with these storage systems too: it just isn't all that obvious until one of the paths fails.

MK

[Ivan Ferreira]

I agree with Matti Kurkela. I think that you need to enable MPIO.

[Danesh Qureshi]

I have found the problem to the device errors I was receiving.

Before I started to install RHEL AS 4 on the server it was running on RHEL AS 3.

From the SAN I unpresented the vdisk to the server but I did not delete the host entry to the server. So later when I installed RHEL AS 4, I represented the vdisk to the server successfully, however because the SAN still had the old entry from when the server was running on RHEL AS 3 it was unable to recognise the server was now running on RHEL AS 4. So what I did from the SAN was to unpresent the vdisk to the server and then deleted the host entry. Then I re-added the host entry and represented the vdisk to the server. I then rebooted the server to allow the changes to take affect. Now I am not seeing any errors when I issue any LVM commands like pvscan, vgscan,pvs,etc. No errors in the messages file or dmesg.

So if you have a server connected to a SAN and you are planning to upgrade the OS on it, you must unpresent and delete the host entry from the SAN before you upgrade your server. After upgrading your server you can then re-add the host entry and re-present the vdisk to the host.

Hope this helps anyone doing a similar excercise.

[skt_skt]

i did not understand the vdisk here? what is that exactly?

[Danesh Qureshi]

I have attached some screen shoots of the Command View. Please have a look at it.

The server/host only started reporting /dev/sda read failures after I had installed RHEL AS 4 as previously it was running RHEL AS 3. When I unpresented the vdisk/luns to the server it was still running on RHEL AS 3, I did not delete the old host entry from Command View.Therefore, the SAN EVA3000 was still recognising the host/server running on RHEL AS 3 even though I had installed RHEL AS 4 and re-presented vdisk/luns to the host/server.

The point to remember is if you are upgrading to a newer OS then you must unpresent the old vdisk and delete the old host entry before upgrading. Once the host/server has been upgraded the you need to re-create a new host entry and then re-present vdisk/luns to the host/server.

Does this makes it clearer.