topic Re: Dealing with stale extents (I/O error) in Operating System - HP-UX

Dealing with stale extents (I/O error)

Mike Keys — Wed, 25 Oct 2006 13:08:53 GMT

I recieved an EMS message stating that the disk at 0/1/1/0.1.0 (c2t1d0) was goin bad. I read through the forums here and did the vgdisplay and found lvol3 to be stale. I then ran the lvdisplay for that lvol and found 2 stale extents. I received a new drive via support and I ran lvdisplay on all lvols for vg00. The mirrored drive 0/1/1/1.0.0 (c3t0d0) had all extents current.

I went ahead and broke the mirrors, reduced the bad disk out of the vg, replace, did the pvcreate, vgextend and mkboot commands. I then started to remirror the lvols. Now, I have three lvols (lvol3, lvol7 and lvol8) that are reporting stale extents and wont sync.

What course of action do I have? Tape restore?

Re: Dealing with stale extents (I/O error)

Steven E. Protter — Wed, 25 Oct 2006 13:15:34 GMT

Shalom,

lvsync sometimes helps. If not, you may need to destroy the logical volumes and restore the data form tape.

Sorry.

SEP

Re: Dealing with stale extents (I/O error)

Mike Keys — Wed, 25 Oct 2006 13:18:34 GMT

I just ran the following command:
echo 2400?20x | adb {device name}
for both disks. Can someone explain the output to me?

This is the output from the drive that had all of its extents current before I broke the mirrors and replaced the "bad" drive.

# echo 2400?20x | adb /dev/dsk/c3t0d0
2400: 4445 4645 4354 3031 0xB0C9 0xF598 8000 0
0xB046 3260 8000 0 0xB0B2 3F60 8000 0
0xB0C6 0xAC60 8000 0

This is the output from the replaced drive and is still reporting stale extents.

# echo 2400?20x | adb /dev/dsk/c2t1d0
2400: 4445 4645 4354 3031 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0

Re: Dealing with stale extents (I/O error)

Mike Keys — Wed, 25 Oct 2006 13:29:13 GMT

I don't get it. If the stale extents were on the bad drive while the mirrored drive was current, why would I see stale extents on the replaced drive?

It seems to me that if everything was good on the mirrored drive, then the extents should sync right back up.

Re: Dealing with stale extents (I/O error)

Mike Keys — Wed, 25 Oct 2006 14:53:21 GMT

Could it be possible that the replacement disk I was sent is also bad? I only ask because before I replaced the bad disk, only lvol3 (/) was stale. Now I have lvol3 (/), lvol7 (/usr) and lvol8 (/var) stale as well.

Re: Dealing with stale extents (I/O error)

florin_10 — Wed, 25 Oct 2006 18:54:35 GMT

What's your patching level like? Is it up to date?
It sounds like a mirroring issue.
flg

Re: Dealing with stale extents (I/O error)

Bill Hassell — Wed, 25 Oct 2006 19:45:58 GMT

If this is just two disks for VG00 and there is a bad spot on one disk, the rest of the extents will remain current until you replace the disk. When you replace the disk, *EVERY* extent on the replacement disk is stale. Now stale does not mean "unreadable data", it means "un-synced data". In other words, you took a disk that was mostly up to date with matching extents and replaced it with a disk where nothing on the disk matches anything on the working disk. So naturally, every lvol is stale. Just run lvsync and it will bring everything back in sync. This is absolutely normal.

NOTE: Since this is VG00, the replaced disk must be restored with boot information too. This is not accomplished with lvsync because the boot area is not part of an lvol.

Re: Dealing with stale extents (I/O error)

Mike Keys — Wed, 25 Oct 2006 20:56:10 GMT

Bill, I did run lvsync.....many times. It will not resync the lvols that are stale. I keep getting an I/O error.

Read my initial post. After replacing the hard drive, I ran pvcreate, vgextend and mkboot.

Re: Dealing with stale extents (I/O error)

Bill Hassell — Wed, 25 Oct 2006 21:06:46 GMT

Sorry, I did not understand that the new disk is reporting I/O errors. There is nothing you can do with LVM to fix I/O errors because they are due to a hardware failure. Is lvsync terminating with an I/O error? If yes, the replacement disk is apparently no good. This should be confirmed in syslog.

Re: Dealing with stale extents (I/O error)

Bill Hassell — Wed, 25 Oct 2006 21:08:07 GMT

Sorry, I did not understand that the new disk is also reporting I/O errors. There is nothing you can do with LVM to fix I/O errors because they are due to a hardware failure. Is lvsync terminating with an I/O error? If yes, the replacement disk is apparently no good. This can be confirmed in syslog.

Re: Dealing with stale extents (I/O error)

Kyris — Thu, 26 Oct 2006 01:25:20 GMT

Mike,

As said STALE does not mean it's bad news in most cases but another way to try if a disk is no good, which I use, is the following command - HPUX-

dd if=/dev/rdsk/c?t?d? of=/dev/null bs=1024 count=1

This may take time to complete but it will indicate a failing disk that the ioscan command may not pick up.

hope this helps.
kyris

Re: Dealing with stale extents (I/O error)

Darrel Louis — Thu, 26 Oct 2006 02:19:07 GMT

Mike,

Have you tried to take the disk out and put it back, might need a re-spin.
Have you tested the disk, to see it's not a DOA.
What's the exact error you get on the 3 lvols?
- Do you've SCSI errors in your syslog?
- What's the result of pvdisplay/diskinfo?

Darrel

Re: Dealing with stale extents (I/O error)

Mike Keys — Thu, 26 Oct 2006 08:06:54 GMT

Here is the output from pvdisplay that I ran on the new disk I just put in.

pvdisplay -v /dev/dsk/c2t1d0|grep stale
00561 stale /dev/vg00/lvol3 00011
01425 stale /dev/vg00/lvol7 00000
01589 stale /dev/vg00/lvol7 00164
01615 stale /dev/vg00/lvol8 00002
01621 stale /dev/vg00/lvol8 00008

The stale extents match up with what I am seeing when I run lvdisplay.

There are no stale extents on the other drive.

Re: Dealing with stale extents (I/O error)

Mike Keys — Thu, 26 Oct 2006 08:08:38 GMT

A 1MB read of the disk shows o.k.

# dd if=/dev/dsk/c2t1d0 of=/dev/null bs=1024 count=1
1+0 records in
1+0 records out

I can run it for he entire disk but it will take up to 2 hours to finish.

Re: Dealing with stale extents (I/O error)

Mike Keys — Thu, 26 Oct 2006 08:26:02 GMT

Well, it finished quicker than I thought.

# dd if=/dev/dsk/c2t1d0 of=/dev/null bs=1024
35566480+0 records in
35566480+0 records out

I'll run against other disk as well and report back.

Re: Dealing with stale extents (I/O error)

Bill Hassell — Thu, 26 Oct 2006 08:43:43 GMT

Just a note about dd. Any blocksize less than 32k will run very slowly. Try this:

dd if=/dev/dsk/c2t1d0 of=/dev/null bs=256k

I think you'll find 64k to 512k significantly faster.