Temperory break mirrored root disk in linux

[Manoj Sivan]

Hi Friends,

I am having one ML370 server running RH Linux AS 3.0 with software mirror in root disk. Today copmpaq performace monitor agent showing predective failiure in disk. So plese advice me how to break the mirror and replace the disk without down time.

Thanks a lot in advance

Manoj

[Ivan Ferreira]

I'm not sure if the disks are hot swap, but if so, this is what you should do:

You can use the following command to remove one member from the mirror:

Assume that you have md0(sda1,sdc1), md1(sda2,sdc2) and md2(sda3,sdc3). And sdc is the disk that you want to replace:

mdadm /dev/md0 â remove /dev/sc1
mdadm /dev/md1 â remove /dev/sc2
mdadm /dev/md2 â remove /dev/sc3

Check the status of the array with cat /proc/mdstat.

Remove the disk, insert the new disk, and runt rescan-scsi-bus.sh (download from internet http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)

Then repartition the new disk exactly as the disk that must be mirrored. Change the partition type to software mirror.

Then run the following commands:

mdadm /dev/md0 â add /dev/sc1
mdadm /dev/md1 â add /dev/sc2
mdadm /dev/md2 â add /dev/sc3

Hope this helps and good luck.

[Sergejs Svitnevs · Accepted solution]

1. Watch the status of your md device:
# mdadm --detail /dev/
example: #mdadm --detail /dev/md0

2. Manually fail a partition, meaning to take it out of the RAID array:
# mdadm /dev/md0 -f /dev/
example: #mdadm /dev/md0 -f /dev/hdb1

3. Replace failed disk

4. Add new disk back to the array
# mdadm /dev/
-a /dev/
example: #mdadm /dev/md0 -a /dev/hdb1

5. Check the status again:
# mdadm --detail /dev/

Regards,
Sergejs

[Manoj Sivan]

Hi Sergejs Svitnevs

Thanx a lot for the suggessions. I missed one thing.only my "/" partition is in the mirror in md0, but /boot is still not mirrored since its is a software raid. I am attaching the output output mdadm --detail for your verification.

Please suggest me how to replace the disk without data loss.

Waiting for your valuable feedback to proceed.

Cheers

Manoj

[Ivan Ferreira]

In the output of mdadm --detail /dev/md0 it does not shows the device that previously conformed the array. Please post the cat /proc/mdstat output.

From the other output, it may be that the device is /dev/sda3. But /dev/sda is also used in the other RAID devices and also is your /boot partition.

BTW, you can mirror the boot partition also.

From that output, it seems that only one partition of /dev/sda detected as failed. I'm afraid that if you replace the device, as your /boot is not mirrored, you may need additional steps to boot the system, and re-install the boot loader. This could make the replacement of the disk not too easy.

[Manoj Sivan]

Hi Ivan,

The failed partition in mdadm --detail /dev/md0 is /dev/sda3. Attaching is the contents of /proc/mdstat file.

[root@fwsvr4 root]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
Event: 2
md0 : active raid1 sdb3[1]
40162432 blocks [2/1] [_U]

md1 : active raid1 sdb4[1] sda4[0]
99032576 blocks [2/2] [UU]

unused devices:
[root@fwsvr4 root]#

============================================
So how can i add boot boot also in another RAID?. How should i proceed in this situation ?

Please help

Thanks in Advance

manoj

[Ivan Ferreira]

This is what would I do. But I want to WARN you that this is my best efford to assist you, I won't warranty that everything works fine. Please use this procedure with caution. I did it several times but is different when you don't touch the system.

To mirror the boot partition:

tar cvf /root/boot.tar /boot
umount /boot
umount /boot.bak

fdisk /dev/sda
Change parititon 1 to software RAID
fdisk /dev/sdb
Change parititon 1 to software RAID

mdadm â create â verbose /dev/md3 â level=1 â raid-devices=2 /dev/sda1 /dev/sdb1

mdadm -As /dev/md3

mkfs -t ext3 /dev/md3
tune2fs -m0 -i0 -c0 /dev/md3
mount /dev/md3 /boot
cd /
tar xvf /root/boot.tar

Edit /etc/fstab and update the new /boot device, remove /boot.bak

Install the boot loader on both disks:

grub-install /dev/sda
grub-install /dev/sdb


I also need to know the output of vgdisplay -v to guide you in the steps to replace the disk.

[Manoj Sivan]

Hi Ivan,

Thannks a lot for the help. Kindly clarify the below commands suggested, i am not very clear.

fdisk /dev/sda
Change parititon 1 to software RAID
fdisk /dev/sdb
Change parititon 1 to software RAID

What option i need to slect to change this ??

mdadm â  create â  verbose /dev/md3 â  level=1 â  raid-devices=2 /dev/sda1 /dev/sdb1

The command is having some jung charactor. not clear.Please post again.

Below is the output of vgdisplay.
=============================================
[root@fwsvr4 root]# vgdisplay -v
--- Volume group ---
VG Name vg00
VG Access read/write
VG Status available/resizable
VG # 0
MAX LV 256
Cur LV 1
Open LV 1
MAX LV Size 2 TB
Max PV 256
Cur PV 1
Act PV 1
VG Size 546.88 GB
PE Size 32 MB
Total PE 17500
Alloc PE / Size 5120 / 160 GB
Free PE / Size 12380 / 386.88 GB
VG UUID KW8Ff2-PGXf-7Kov-XwWv-SpNJ-K30m-SGbwzI

--- Logical volume ---
LV Name /dev/vg00/lvol1
VG Name vg00
LV Write Access read/write
LV Status available
LV # 1
# open 1
LV Size 160 GB
Current LE 5120
Allocated LE 5120
Allocation next free
Read ahead sectors 1024
Block device 58:0


--- Physical volumes ---
PV Name (#) /dev/cciss/c0d0p1 (1)
PV Status available / allocatable
Total PE / Free PE 17500 / 12380



[root@fwsvr4 root]#

============================================

Thanks and regards

Manoj

[Ivan Ferreira]

To change the partition type, use the t command in fdisk (toogle), and select fd as the code.

fdisk /dev/sda
t
1
fd

fdisk /dev/sdb
t
1
fd

The command should be:

mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

To replace the disk in your current configuration:

Considering the output of your commands, to replace /dev/sda you should stop using the device:

1- Umounting /boot. You must save the content of this partition to restore after the disk replacement because this partition is not on a raid device.
umount /boot

2- Manually remove sda4 from /dev/md1.
mdadm /dev/md1 --remove /dev/sda4

Your volume group is on a hardware raid device so nothing has to be done there.

3- If you will use the same disk model, save the current partition table.
sfdisk â d /dev/hda > /tmp/partitions

4- Remove the disk (if hot swap capable), insert the new disk and run rescan-scsi-bus.sh

5- Restore the partition table.
sfdisk /dev/hdc < /tmp/partitions

6- Create a file system for /boot.
mkfs -t ext3 /dev/sda1

7- Restore /boot.
cd /
tar xvf /root/boot.tar

8- Install the boot loader.
grub-install /dev/hda

9- Add the devices to their respective raid device.

mdadm /dev/md0 --add /dev/sda1
mdadm /dev/md1 --add /dev/sda4

Please remember to assign points if this helps. That motivates me.

Good luck.