Operating System - HP-UX
1833764 Members
2602 Online
110063 Solutions
New Discussion

Dealing with stale extents (I/O error)

 
Mike Keys
Regular Advisor

Dealing with stale extents (I/O error)

I recieved an EMS message stating that the disk at 0/1/1/0.1.0 (c2t1d0) was goin bad. I read through the forums here and did the vgdisplay and found lvol3 to be stale. I then ran the lvdisplay for that lvol and found 2 stale extents. I received a new drive via support and I ran lvdisplay on all lvols for vg00. The mirrored drive 0/1/1/1.0.0 (c3t0d0) had all extents current.

I went ahead and broke the mirrors, reduced the bad disk out of the vg, replace, did the pvcreate, vgextend and mkboot commands. I then started to remirror the lvols. Now, I have three lvols (lvol3, lvol7 and lvol8) that are reporting stale extents and wont sync.

What course of action do I have? Tape restore?
15 REPLIES 15
Steven E. Protter
Exalted Contributor

Re: Dealing with stale extents (I/O error)

Shalom,

lvsync sometimes helps. If not, you may need to destroy the logical volumes and restore the data form tape.

Sorry.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

I just ran the following command:
echo 2400?20x | adb {device name}
for both disks. Can someone explain the output to me?


This is the output from the drive that had all of its extents current before I broke the mirrors and replaced the "bad" drive.

# echo 2400?20x | adb /dev/dsk/c3t0d0
2400: 4445 4645 4354 3031 0xB0C9 0xF598 8000 0
0xB046 3260 8000 0 0xB0B2 3F60 8000 0
0xB0C6 0xAC60 8000 0

This is the output from the replaced drive and is still reporting stale extents.

# echo 2400?20x | adb /dev/dsk/c2t1d0
2400: 4445 4645 4354 3031 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

I don't get it. If the stale extents were on the bad drive while the mirrored drive was current, why would I see stale extents on the replaced drive?

It seems to me that if everything was good on the mirrored drive, then the extents should sync right back up.
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

Could it be possible that the replacement disk I was sent is also bad? I only ask because before I replaced the bad disk, only lvol3 (/) was stale. Now I have lvol3 (/), lvol7 (/usr) and lvol8 (/var) stale as well.
florin_10
Frequent Advisor

Re: Dealing with stale extents (I/O error)

What's your patching level like? Is it up to date?
It sounds like a mirroring issue.
flg
Bill Hassell
Honored Contributor

Re: Dealing with stale extents (I/O error)

If this is just two disks for VG00 and there is a bad spot on one disk, the rest of the extents will remain current until you replace the disk. When you replace the disk, *EVERY* extent on the replacement disk is stale. Now stale does not mean "unreadable data", it means "un-synced data". In other words, you took a disk that was mostly up to date with matching extents and replaced it with a disk where nothing on the disk matches anything on the working disk. So naturally, every lvol is stale. Just run lvsync and it will bring everything back in sync. This is absolutely normal.

NOTE: Since this is VG00, the replaced disk must be restored with boot information too. This is not accomplished with lvsync because the boot area is not part of an lvol.


Bill Hassell, sysadmin
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

Bill, I did run lvsync.....many times. It will not resync the lvols that are stale. I keep getting an I/O error.

Read my initial post. After replacing the hard drive, I ran pvcreate, vgextend and mkboot.
Bill Hassell
Honored Contributor

Re: Dealing with stale extents (I/O error)

Sorry, I did not understand that the new disk is reporting I/O errors. There is nothing you can do with LVM to fix I/O errors because they are due to a hardware failure. Is lvsync terminating with an I/O error? If yes, the replacement disk is apparently no good. This should be confirmed in syslog.


Bill Hassell, sysadmin
Bill Hassell
Honored Contributor

Re: Dealing with stale extents (I/O error)

Sorry, I did not understand that the new disk is also reporting I/O errors. There is nothing you can do with LVM to fix I/O errors because they are due to a hardware failure. Is lvsync terminating with an I/O error? If yes, the replacement disk is apparently no good. This can be confirmed in syslog.


Bill Hassell, sysadmin
Kyris
Frequent Advisor

Re: Dealing with stale extents (I/O error)

Mike,

As said STALE does not mean it's bad news in most cases but another way to try if a disk is no good, which I use, is the following command - HPUX-

dd if=/dev/rdsk/c?t?d? of=/dev/null bs=1024 count=1

This may take time to complete but it will indicate a failing disk that the ioscan command may not pick up.

hope this helps.
kyris
Darrel Louis
Honored Contributor

Re: Dealing with stale extents (I/O error)

Mike,

Have you tried to take the disk out and put it back, might need a re-spin.
Have you tested the disk, to see it's not a DOA.
What's the exact error you get on the 3 lvols?
- Do you've SCSI errors in your syslog?
- What's the result of pvdisplay/diskinfo?


Darrel
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

Here is the output from pvdisplay that I ran on the new disk I just put in.

pvdisplay -v /dev/dsk/c2t1d0|grep stale
00561 stale /dev/vg00/lvol3 00011
01425 stale /dev/vg00/lvol7 00000
01589 stale /dev/vg00/lvol7 00164
01615 stale /dev/vg00/lvol8 00002
01621 stale /dev/vg00/lvol8 00008

The stale extents match up with what I am seeing when I run lvdisplay.

There are no stale extents on the other drive.
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

A 1MB read of the disk shows o.k.

# dd if=/dev/dsk/c2t1d0 of=/dev/null bs=1024 count=1
1+0 records in
1+0 records out

I can run it for he entire disk but it will take up to 2 hours to finish.
Mike Keys
Regular Advisor

Re: Dealing with stale extents (I/O error)

Well, it finished quicker than I thought.

# dd if=/dev/dsk/c2t1d0 of=/dev/null bs=1024
35566480+0 records in
35566480+0 records out

I'll run against other disk as well and report back.
Bill Hassell
Honored Contributor

Re: Dealing with stale extents (I/O error)

Just a note about dd. Any blocksize less than 32k will run very slowly. Try this:

dd if=/dev/dsk/c2t1d0 of=/dev/null bs=256k

I think you'll find 64k to 512k significantly faster.


Bill Hassell, sysadmin