Replacing a boot disk WITH Mirroring

[Indira Aramandla]

I have a query in regards to “Replacing a boot disk WITH Mirroring”.

If one server has a faulty internal disk, and the disk being replaced is Hot-Pluggable (or Hot-Swappable), does the meaning of Hot-Pluggable apply only to hardware where you can pull the disk out on-line and replace with the new one. But the from the softwere will Hot-Pluggable mean anythign or do we still need to lvreduce the mirror for all the logical volumes defined on the vg00 before repalcing the disk.

After replacing the internal disk we have to do a vgcfgrestore, vgchange, make it a boot disk, (update the diagnostics if you have installed ), verify that the BDRA information is accurate and then syncronize the BDRA with vgsync vg00. This bit is fine.

Last week we had a situation like this.

We have an L-class server (9000/800/L1500-7x) rp5430 which has two internal disks (HP36.4GST336753LC) that are mirrored.

One of the disks has become faulty and it raised a smar event. Resmon summary says “Disk at hardware path 0/0/1/1.2.0 : I/O request failed.”, where the harware paht refers to this faulty internal disk. When we do ioscan the disk shows as claimed. But vgdisplay of vg00 shows the logical volumes as slate and it clearly shows that we are unable to write on to that disk. So the engineer came with the new disk and before repalcing the disk he issued lvreduce for all the logical voulmes on vg00. And when it was reducing lvol7 which is /usr the system went into panic mode and re-booted itself.

So to the system came backup, we had lvol1, lvol2….lvol6 reduced successfully and lvol7, lvol8, lvol9 (/usr, /var, /opt) were not reduced and were left as stale. We replaced the disk and when tried to do vgcfgrestore –n it gave us errors. And then we discovered that /etc/lvmconf/vg00.conf was over writtena nd was zero bytes. We restored from backup and had to issue vgcfgrestore with a –R option.

Here are my queries.

1. Does Hot-Pluggable means only form the hardware and not from the software, where when you just remove the disk and replace with the new disk and do vgcfgrestore the system software will not worry about the previous mirror and restore to the new disk.

2. When can you say the disk is dead. If ioscan shows that then fine. But some times the disk has some stale sectors and has problmes writing on to it, but the OS still says the disk as claimed. In that case is it advisable to just remove the disk without reducing the mirror.

3. In out case was lvreduce a necessary step, ot would we had got away with replace the disk straight away and do vgcfgrestore.

Any suggestions……/ explainations…… much appreciated.

In

[Steven E. Protter]

It can be claimed on ioscan and dead. You have seen this situation yourself with this sitaution. If resmon/ems says its dead andyou can't write, its dead.

I had a HP-6000 disk array on my D class box(basement experiment) which mirrored an entire system fail recently.

I ended up breaking the mirrors that survived this one disk failure nad doing a vgreduce -f and following the instructions afterwards to have a working unmirrored system.

If the disk is dead, you probably won't be able to break the mirror, but if you can break the mirror it is advisable for you to do so.

SEP

[Indira Aramandla]

Hi Steven,

Thanks for the quick reply.

Now I understand that even if the ioscan shows as claimed, if resmon/ems or syslog says it is dead and we cannot write to it then the disk is dead.

I do understand that the properway would be to break the mirror and then replace. But as we experienced a system panic (this was the commnad used to recude the mirror--- for i in lvol1 lvol2 lvol3 lvol4 lvol5 lvol6 lvol7 lvol8 lvol9
do lvreduce -m 0 /dev/vg00/$i /dev/dsk/c1t2d0
done).

So I was thinking with hot-pluggable disks can we get away with not reducing the mirror and repalce then vgcfgrestore.

Indira A

[Ermin Borovac]

I think correct command would have been

# lvreduce -A n -m 0 ...

Option '-A n' is to prevent automatic vgcfgbackup operation, which is likely to run into problems when accessing defective disk.

Reducing mirrors is recommended if defective disk is considered to be attached (i.e. pvdisplay command returns available/unavailable status). If pvdisplay returns error then disk is unattached; you can proceed with vgcfgrestore without reducing mirrors.

[Rajeev Shukla · Accepted solution]

1. It depends Hot-Pluggable also sometimes means that you dont have to worry about the software, but it depends how intelegent the software is. In case of disk arrays having RAID, which is pure example of Hot-Pluggable, where you dont worry about restoring the data on the replaced disk. But Mirror/UX is not so smart to do things my itself. Actually it is smart, you dont have to split mirrors or make disk bootable etc, Just do vgcfgrestore on the new disk and vgsync or vgchange will mirror the entire disk.

2. I have seen it works even in case the disk is not totally bad, you just replace the disk, do vgcfgrestore, vgchange and vgsync
3. In your case too i suppose you could have saved time by just replacing the disk and performing the above steps.

Cheers
Rajeev

[Sudeesh]

Hi,
Almost all the systems in our env have root mirroring and we face such situations often.
These are the steps I follow to replace root mirror disk.

Incase disk is offline I will first do
#ioscan -H
This will try to bring the disk online. This ensures the disk is really dead.

If disk is claimed and still having issues, do a diskinfo.
#diskinfo -v /dev/rdsk/c0t5d0
in case of bad disk this command will give error or will report size as zero.

You can also use dd to check the disk media.

All the above steps go through and still you have stale extends, go ahead and replace disk. Before that ensure only one disk have stale extends with lvdisplay.
#lvdisplay -v /dev/vg00/lvol*

At this point i will take out the faulty disk.

Insert new disk.

#ioscan -H
To put the new disk as claimed from NO_HW.

#mv /etc/lvmtab /etc/lvmtab.orig
If entry for specified disk is already there in lvmtab, pvcreate will fail.

#pvcreate -B /dev/rdsk/c0t5d0
#mv /etc/lvmtab.orig /etc/lvmtab
#vgcfgrestore -n vg00 /dev/rdsk/c0t5d0
#vgchange -a y /dev/vg00
#vgsync vg00
if disk does not automatically start syncing

#mkboot /dev/rdsk/c0t5d0
#mkboot -a "hpux -lq" /dev/rdsk/c0t5d0
#lvlnboot -v

Issue lvdisplay -v /dev/vg00/lvol* to ensure all extends are current.


As far as your first question is concearned I belive hardware,Driver and OS should support to make a device Hot-Pluggable.

Thanks,

Sudeesh

[Indira Aramandla]

Thanks Rajeev.

It is a lot clear now.

It does make sence when you say that it depends on "how intelegent the software is"
.
You mentioned, "Mirror/UX is not so smart to do things by itself. Actually it is smart, you dont have to split mirrors or make disk bootable etc"

Does that mean Mirror/UX is not capable of and we have to do vgcfgrestore and vgchange or vgsync and make it bootable.

In our case the disk was not totally bad, it had a few stale sectors. So you recon we had just to replace the disk and perform vgcfgrestore and vgchange. What about making it bootable and updating the disgnostics and syncronize the BDRA.

Thanks
Indira A

[Indira Aramandla]

Thanks Ermin,
That commnad lvreduce was performed by HP field Engineer on site.

Thanks Sudeesh, for your detailed steps. It is a lot clear now what hot-pluggable means. And when the disk is dead then one can replace it straight away and perform vgcfgrestore and vgchange...., without reducing the mirror.

Thanks
Indira A

[Sudeesh]

Use following steps to update ODE.

#cd /usr/sbin/diag/lif
#mkboot -b updatediaglif -p ISL -p HPUX -p LABEL -p AUTO /dev/rdsk/c0t5d0


Sudeesh

[Indira Aramandla]

Hi Sudeesh,

Yes that is how we did to update ODE.

Use mkboot to add boot information,
# mkboot /dev/rdsk/c0t6d0
# mkboot -a "hpux (;0)/stand/vmunix" /dev/rdsk/c1t2d0

# cd /usr/sbin/diag/lif
# mkboot -b updatediaglif -p ISL -p AUTO -p HPUX -p LABEL -p PAD /dev/rdsk/c1t2d0

Indira A

[Rajeev Shukla]

Hi Indira,

By saying that Mirror/UX is not that smart, i meant that..smartness would have been if it could identify the new disk (after replacement) and perform the necessary steps as data replications and sync VGRA.

Sorry for not putting all commands in sequence and as you mentioned in u'r previous post you do need to make the disk bootable and copy over the ODE to new disk.

Cheers
Rajeev

[Indira Aramandla]

Thanks Rajeev,

Yes it would had been a lot simpler if Mirror/UX software was that smart enough to do all those things.

Thanks for your hlep.
Indira A

[Indira Aramandla]

Closing this thread as my doubts have been clarified.


Indira A