I/O error on iSCSI LUN - no errors on SAN

[Charles Slivkoff]

There was a significant network outage which caused a loss of contact to 2 iSCSI volumes. These are both managed by LVM. One of them I had no problems with after a fsck.

The other, though, is refusing to fsck. I noticed there are I/O errors being reported. Both of these volumes are on the same SAN array.

We had the SAN admin clone the volume and present the clone to this server, but we get the same problems.

$ fsck -F vxfs -o full -n /dev/clone/lvol1
pass0 - checking structural files
pass1 - checking inode sanity and blocks
fileset 999 primary-ilist inode 215408 has invalid number of blocks (73324)
fileset 999 primary-ilist inode 215408 failed validation clear? (ynq)n
UX:vxfs fsck: ERROR: V-3-25432: fsck read failure devid = 0, bno =
8520128, off = 0, len = 65536

A "dd" from the lvol fails with an I/O error (after the LV Timeout is reached). A "dd" from the dsk device blocks indefinitely and I was forced to reboot.

The server is running 11.23 with the iSCSI software initiator, version B.11.23.03e. I noticed an update to this on software.hp.com and am going to try it, but I am not optimistic.

The error seems to imply a "hw" issue, but that is not possible since there are no SAN errors and the problem has been replicated with a clone of the volume. This behavior just does not make any sense.

Does anyone have any suggestions?

[TTr]

> We had the SAN admin clone the volume

How was it cloned? Some storage systems have different levels and methods of cloning and sometimes the clone really points back to the original, specially since there have been no updates to your original LUN, the clone would completely point to it.

Making a full LUN copy would be more conclusive or if the SAN admin has another tool, read the entire LUN for errors. Don't rely on the fact that the LUN is there and has no errors reported within the SAN.

[Charles Slivkoff]

Thanks, I had not thought of the "clone" being "sparse" and some sort of "copy on write" clone. I'm checking on that now.

[Charles Slivkoff]

The iSCSI update B.11.23.03f had no effect on the problem.

[Charles Slivkoff]

The "clone" was a "copy on write". By writing over the "bad" blocks, the "clone" behaved normally.

The SAN vendor is looking at a problem on their end.

[Charles Slivkoff]

Problem with SAN.