Stale in the secondry disk but primary disk has gone bad

[Arunkumar.B]

Hi Experts,

I have two root disk mirrored.We brought into notice the root filesystem in the Alternate disk is stale.We were pointing the alternate disk like dd.But the ALternate disk is fine

After some times we come to know that the Primary disk is having bad block.


Now i need to change the Primary disk.The root Filesystems in the Alternate disk having 4 stales.I tried to take ignite today evening for changing the Primary disk but its giving I/O error.Backup got aborted.

U pls people suggest me how to go about this without losing my data.The server is working fine now...But i need to change the Primary root disk now.

1)I can't boot through Alternate disk bcoz its having stale in root
2)I can't take Ignite backup bcoz its giving I/O error


Pls Suggest me a way out ...

[A. Clay Stephenson]

Your first task should be an lvsync for each stale LVOL.

[Arunkumar.B]

Hi Clay,

I tried Vgsync & lvsync

It saying I/O error.

Thanx in Advance
Arunkumar.B

[A. Clay Stephenson]

Bear in mind that the / filesystem is all but static so that even with stale extents there is a very high probability that you are actually ok. You have very little to lose right now so I would yank the failed primary disk and proceed with the normal restore of a mirrored root disk. If you have hot-plug disks then you shouldn't need to shutdown.

[Arunkumar.B]

HI Clay,

I agree to ur point.I can remove the primary disk & start mirroring from the alternate disk with the new disk.But the thing is 4 stales in the root are showing in Alternate disk.After Removing or Mirroring the new disk with Alternate disk which is having stale .If its gone down then i will be in soup...

[Arunkumar.B]

Hi Clay,

Is there any other way ...

[A. Clay Stephenson]

Typically, your situation arises when a failed disk goes too long undetected. At the very least you should already have had an Ignite image before you actually need it -- so I don't have much sympathy about your failed Ignite effort. If you have good backups (including Ignite's) then you don't need them; if you don't have good backups ... it's one of God's little jokes.

In my case, I always hacve good backups, Ignites, plus lifeboats -- which are raw copies of boot disk(s) made while the machine is running (i.e. dirty filesystems). The important point about lifeboat disks is that I have used them many times and those are about as stale a copy as one could have and yet I have never had one fail to boot a machine. They do have to perform an fsck, of course, because the filesystems were not unmounted at the time of the copy.

My next attempt would be to pull your failed disk out a few centimeters, let it spin down while still in the slot, and then reinsert it. It may come back to life itself - then do an lvsync. If this effort fails then yank it and replace and proceed with the normal restore. Even if the original disk does revive itself, you should replace it.

[Florian Heigl (new acc)]

While lvols do show the stale PEs?

Can You isolate the SCSI error messages in syslog that led to those four stale PE?

[Darrel Louis]

Hi,

Do you have the possibillity to add another disk in the machine or a free disk, if yes you can start a mirror to this disk and see if the mirror goes oke.

If you have a good Ignite backup tape you shouldn't worry at all.

Darrel

[Albert_31]

Hello Arun,

Depending on which logical volume you are getting this stale erros.. since some of the LV's have bad block reallocation turned off..There is a way out of the situation and carries with it a risk for data loss..

HP has a tool which helps in clearing this error..something which I am not clear about but helps..you can try that.

[Michael Steele_2]

Along as you stay within HW classes cloning works just fine. This involves make_tape_recovery from another server identical to your failed one.


a) create and verify an ignite tape from another server.
b) backup data on bad server
c)Collect all your ioscans, boot disks, O/S file system sizes, hostname, ip, gateway, ntp, nis, swap sizes, RAM, etc.
d) replace bad disks
e) Boot off of a other server's tape.

Interrupt at 10 second prompt. (* don't opt for automatic recovery *)

Advanced installation.

Use tab, enter and arrow keys. Fill in all the data fields. Modify with information collected in step c)

[Mridul Shrivastava]

I feel you have nothing to loose and can go ahead with replacing primary disk.
you are seeing 4 stale extents on secondary disk because of the primary disk gone bad and it can't sync the data from primary disk so whatever is there is intact.
So actually the data at secondary disk is absolutely fine, it is showing as stale as it can't sync those with primary (because of HW issues on primary).
I would suggest you to replace the primary disk as early as possible to avoid any issues in future.

[Arunkumar.B]

HI,

The Case is closed.This is a superdome server(VPAR).Customer is taking backup through ignite server(one of the vpar)Luckily he had an archive of two month's back.We restored through that one & CLuster ,server come up.Working Fine..


Thanks For all your Contributions

Cheers
Arunkumar.B

[Arunkumar.B]

Hi All,

I am Closing the Query


With Cheers
Arunkumar.B