HPUX can't boot automatically

[Adisuria Wangsadinata_1]

Hi Just_K,

Looks like one of your harddisk is faulty already, that's why there's an error message when you did a vgsync : 'Couldn't re-synchronize stale partitions of the logical volume: I/O error'

You can check on the output of the last dd command, with this you can know which one is a faulty harddisk.

If you have an Ignite/UX backup, I refer you to prepare it. Since it's better & fastest way to restore from it, since we're not sure whether the other disk is ok.

BTW, may i know what's the result of dd command ? If hang, may i know on which harddisk ?

Cheers,
AW

[freddy_21]

please check lvdisplay -v /dev/vg00/lvol8

you can found which disk has fault.

ex:
0025 /dev/dsk/c1t6d0 1500 current /dev/dsk/c1t5d0 1500 current
0026 /dev/dsk/c1t6d0 1501 current /dev/dsk/c1t5d0 1501 current
0027 /dev/dsk/c1t6d0 1502 current /dev/dsk/c1t5d0 1502 current
0028 /dev/dsk/c1t6d0 1503 current /dev/dsk/c1t5d0 1503 current
0029 /dev/dsk/c1t6d0 1504 current /dev/dsk/c1t5d0 1504 current
0030 /dev/dsk/c1t6d0 1505 current /dev/dsk/c1t5d0 1505 current
0031 /dev/dsk/c1t6d0 1506 current /dev/dsk/c1t5d0 1506 current
0032 /dev/dsk/c1t6d0 1507 current /dev/dsk/c1t5d0 1507 current
0033 /dev/dsk/c1t6d0 1508 current /dev/dsk/c1t5d0 1508 current
0034 /dev/dsk/c1t6d0 1509 current /dev/dsk/c1t5d0 1509 current
0035 /dev/dsk/c1t6d0 1510 current /dev/dsk/c1t5d0 1510 stale


it means your disk c1t5d0 have problem. replace it.

thanks
freddy

[Just_K]

PV2 PE2 Status 2
/dev/dsk/c4t0d0 01599 stale
/dev/dsk/c4t0d0 01600 current
/dev/dsk/c4t0d0 01601 stale

which commands i need to run after disk replacing?

[Adisuria Wangsadinata_1]

Hi Just_K,

Check also whether you have 'stale' status also in other disk (c2t0d0)

# lvdisplay -v /dev/vg00/lv* | grep stale

If just c4t0d0, I refer you to call HP to replace the harddisk.

If your harddisk is not a hot plugable harddisk, here is the document for steps to replacing a mirrored root disk (docID : LVMKBRC00006116) :

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000068955762

Hope this information can help you.

Cheers,
AW

[freddy_21]

it meas your disk have problem. from output lvdisplay, i saw your configuration not have mirror. i think prepare your backup OS.


Thanks
freddy

[Just_K]

here is full lvdisplay output:

bash-2.04# lvdisplay -v /dev/vg00/lvol8
--- Logical volumes ---
LV Name /dev/vg00/lvol8
VG Name /dev/vg00
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 1928
Current LE 241
Allocated PE 482
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c2t0d0 241 241
/dev/dsk/c4t0d0 241 241

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c2t0d0 01599 current /dev/dsk/c4t0d0 01599 stale
00001 /dev/dsk/c2t0d0 01600 current /dev/dsk/c4t0d0 01600 current
00002 /dev/dsk/c2t0d0 01601 current /dev/dsk/c4t0d0 01601 stale

[Adisuria Wangsadinata_1]

Hi Just_K,

Did your c2t0d0 have 'stale' status also ?
If c2t0d0 dont have stale status, it means c2t0d0 is working fine.

All you have to do is to replace c4t0d0 only, if only c4t0d0 with stale status.

If you want, just attach the output from the command below :

# lvdisplay -v /dev/vg00/lv*

Hope this information can help you.

Cheers,
AW

[Just_K]

Only c4t0d0 have 'stale' status. I'm going to replace it.
All results of replacing will be here.

Thanx for help.

[Just_K]

Only c4t0d0 have 'stale' status. I'm going to replace it.
All results of replacing will be here.

Thanx for help.

[Adisuria Wangsadinata_1]

Hi Just_K,

If you still need wait for a harddisk to come to your site, you can boot your system on c2t0d0 from now.

Since you're sure that only c4t0d0 with stale status, you are safely to remove the mirror first (c4t0d0) and boot from c2t0d0.

But if the harddisk will come immediate, you can use the steps to replacing a mirrored root disk (docID : LVMKBRC00006116) :

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000
068955762

Hope this information can help you.

Cheers,
AW

[Just_K]

after disk replacing and "vgsync" nothing changed:
bash-2.04# lvdisplay -v /dev/vg00/lv* | grep stale
LV Status available/stale
00000 /dev/dsk/c2t0d0 01599 current /dev/dsk/c4t0d0 01599 stale
00002 /dev/dsk/c2t0d0 01601 current /dev/dsk/c4t0d0 01601 stale

[freddy_21]

yes you must run vgcfgrestore
and it will be automatic sync

if nothing happen, please run vgsync command.

please check your disk led. is it blinking or not

[Just_K]

bash-2.04# vgcfgrestore -n /dev/vg00 /dev/rdsk/c4t0d0
Volume Group configuration has been restored to /dev/rdsk/c4t0d0
bash-2.04# vgsync vg00
Resynchronized logical volume "/dev/vg00/lvol1".
Resynchronized logical volume "/dev/vg00/lvol2".
Resynchronized logical volume "/dev/vg00/lvol3".
Resynchronized logical volume "/dev/vg00/lvol4".
Resynchronized logical volume "/dev/vg00/lvol5".
Resynchronized logical volume "/dev/vg00/lvol6".
Resynchronized logical volume "/dev/vg00/lvol7".
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgsync: Couldn't resynchronize logical volume "/dev/vg00/lvol8".
vgsync: Couldn't resynchronize volume group "vg00".
bash-2.04#

LED is blinkig when resynchronizing

[Adisuria Wangsadinata_1]

Hi Just_K,

when you run the dd command, is the process hang also on c2t0d0 ?

Or is there any I/O errors comes from c2t0d0 ?

If that so, looks like c2t0d0 need to replace also.

Do you have any Ignite/UX backup for this system ?

if you have, it's time to use it.

Hope this information can help you.

Cheers,
AW

[freddy_21]

before restore, please do lvsync for lvol7 and lvol8.

freddy

[Just_K]

There is no errors on disks at all. It seems that all disks is ok. I think it is problem in mirroring. What if i delete disk c4t0d0 from mirror and after that add it again?
lvsync lvol8 gives I/O error.

[Adisuria Wangsadinata_1]

Hi Just_K,

1. prepare the new disk for c4t0d0
2. remove c4t0d0 from the mirror
lvreduce, vgreduce
3. shutdown the system
4. replace the harddisk c4t0d0 with the new one
5. follow the steps on the document that I gave it to you on previous thread

You need a new disk to replace c4t0d0 since you found stale status on c4t0d0.

At this point, I asume you didnt found any I/O error from dd command & stale status from c2t0d0. So c2t0d0 can consider a good disk.

Hope this information can help.

Cheers,
AW

[Just_K]

same results...
system can't sync lvol8 because of I/O error, but sucsessfully mount it on /var. I thought that if lvol8 can be mounted, then there is no errors. I don't understand it :(

[SVB]

First things first.

Make sure you have at least 1 phsyically working root disk.

Do the command:

# dd if=/dev/dsk/c2t0d0 of=/dev/null bs=4096k

This will take a while. It will try to read every block on your c2t0d0 disk, if the command completes succesfully, this disk if physically ok.

If the command is succesful, it should say something like:

112354+0 records in
112354+0 records out

If it fails, it should say something else, probably: I/O error

Do the same for the other disk:

# dd if=/dev/dsk/c4t0d0 of/dev/null bs=4096k

Again, if this command completes succesfuly, the disk is physically ok.

If both commands are succesful, you don't have a hardware problem with your disks, and you need to start looking at filesystem problems.

If one of these commands isn't succesful, you are likely to have a faulty disk.

If both these commands aren't succesful, I hope you have a good backup...

Anyway, do these 2 commands and post the results here.

[Just_K]

bash-2.04# dd if=/dev/dsk/c2t0d0 of=/dev/null bs=4096k
8683+1 records in
8683+1 records out
bash-2.04# dd if=/dev/dsk/c4t0d0 of=/dev/null bs=4096k
8683+1 records in
8683+1 records out
bash-2.04#

[F Verschuren]

after disk replacing and "vgsync" nothing changed:
bash-2.04# lvdisplay -v /dev/vg00/lv* | grep stale
LV Status available/stale
00000 /dev/dsk/c2t0d0 01599 current /dev/dsk/c4t0d0 01599 stale
00002 /dev/dsk/c2t0d0 01601 current /dev/dsk/c4t0d0 01601 stale


It can be that disk /dev/dsk/c2t0d0 s broken and that the mirror just copyed the bad files to the other part, When I read this story I think that disk /dev/dsk/c2t0d0 needs to be replaced, if you are lukky after that everytin works fine again but it can be that you have to restore the lvol that have the stale blocks.

[SVB]

The output of the dd commands shows that both disks are fine from a hardware point of view.

You need to be very careful. It is quite possible that you have some kind of filesystem problem, that cause the vgsync to fail. To check this, you could try to see if

dd if=/dev/vg00/lvol8 of=/dev/null bs=4096k

is succesfull or fails. If it fails, you have a problem with the filesystem. If it's successful, you can try to remove the dev/dsk/c4t0d0 from the volumegroup and then add it again, recreating the mirrors.

Whatever you do, make sure you have a good ignite tape of your system!

Also, if you're in doubt, and especially if the dd command from above fails, log a call with HP.