reboot caused vg to disappear

[FLQ]

Hi all, I had a little problem with one of my L-Class server.

Here is the desc.

L-2000 2 cpus
1,5 GB RAM
2 FC hooked up to a brocade
1 LUN defined for that machine with 2 paths (c20t0d2,c27t0d2) holding SAP DB on a VA7100.

I needed to reboot the box and I did not make any changes to that box concerning the VA but when it came back up, I had lost the vg pointing to the VA resulting in a down DB... :-((

I did try to mount the FS on VA by hand but cannot query physical device.
I did a vgexport -v -p -s -m /tmp/that_va.map that_va = success

I tried then to vgexport for real = success.

I then tried to vgimport that_va using that_va.map but it could not find anything on the disk devices.

At that point I had to do a vgcfgrestore to make it work and it did, I was able to mount the FS.


My question would be simple...How come I lost that_va when I rebooted?

What might cause that?

Where should I look for clues of what happened?

TIA

Francis

[Martin Johnson]

do "ioscan -fn" and see if everything shows up and is CLAIMED. If there are things missing check the brocade switch.

HTH
Marty

[Stefan Farrelly]

If you say you couldnt query the physical device then its changed. This is very unlikely to have happened on the HP, more likely on the remote disk device or whatever controls the brocade switch.

So your old LUN with c20t0d2 and c27t0d2 now have different device names in your recovered VG ?

[PIYUSH D. PATEL]

Hi,

Check

#strings /etc/lvmtab and see the contents and check for c20t0d2,c27t0d2

Do an ioscan and then check out whther these device files are there and claimed.

Piyush

[George Abraham_1]

hai

I belive you need some patches... have the latest Hardware extension patches?

regards

[FLQ]

I did an ioscan and devices were showing up as CLAIMED

The vg is still on the same devices. c20t0d2 c27t0d2

But I could not query the devices when I did a vgimport
even a vgscan did not find the headers....

Thanks

[FLQ]

Unless it fixes that exact problem and it is stated in the text of the patch, our policy is to not apply patch bundles without first testing it.

That said, I do not have the latest HW bundle installed

[PIYUSH D. PATEL]

Hi,

mv /etc/lvmtab /etc/lvmtab.old
vgscan -a

Was your system running properly before and was this the fisrt time you had this problem ??

You can also load the latest patch bundle on your server to prevent such problems in the future.

Piyush

[FLQ]

The system was working fine before I had to reboot.

I did mv /etc/lvmtab /etc/lvmtab.old

I did vgscan but it did not return the PV that belonged to that_va

[Stefan Farrelly]

If your'e still on the same phyiscal devices and these same devices werent visible when you rebooted then only 2 possibilities;

1. Had the server finished rebooting when you logged into it and checked if the VG's were mounted ? Next time you boot your server log in from another server as soon as you can ping it and do an ioscan -fkCdisk and see if your 2 physical devices are CLAIMED or not, this will tell you if they are being slow in responding to the HP and point to the fibre switch or disk unit having the problem.

2. It could be a problem with the fibre connection to the disk unit. Either the fibre connections arent good, or there are too many devices on the fibre switch (causing a delay in the HP seeing the drives) or a problem on the disk unit. Any of these will cause your symptoms - initially the physical disks arent available but a little while later you can use them.

If you do a full ioscan -fkCdisk how long does it take ? This is a good indiciation of how fast your fibre connections are - it should come back withing 40 secs max. If it takes a lot longer then you have a fibre connection problem.

[James R. Ferguson]

Hi Francis:

This would appear to have been a transient disconnection. I presume that your filesystem is intact, since you don't indicate otherwise. Assuming no corruption of the LVM headers, then it is odd that you couldn't query them with 'vgimport'. 'vgcfgrestore' would only replace LVM information.

It would be (and would have been) interesting to query the physical volume with 'diskinfo':

# diskinfo -v /dev/rdsk/cXtYdZ

Regards!

...JRF...

[Martin Johnson]

Do you have STM installed on the system? You could run it and see if it logged any errors.

HTH
Marty

[FLQ]

Stefan, a full ioscan did not take longer than usual and devices were CLAIMED. I also did a rmsf -H 0/4 0/7 to remove special files then a insf -e to recreate them. It behaved as expected.

James, a diskinfo /dev/[r]dsk/dev/c20t0d2 returned ONLINE status without errors also fcmsutil /dev/td[01], returned also an ONLINE status and no errors.

Thanks for prompt relies

[James R. Ferguson]

Hi (again) Francis:

By the way, did /var/adm/syslog/syslog.log (or /var/adm/syslog/OLDsyslog.log) yield any insight?

Regards!

...JRF...

[Stefan Farrelly]

This is going to be hard to diagnose now without another reboot, or a few reboots.

What I think happens is this;

1. HP reboots.
2. Fibre card activates (link light comes on) once the HP is loading vmunix (not before) as it needs to load the fibre driver in the kernel first.
3. You should watch the link light on your fibre card while it is rebooting to note when it comes on. It may not be coming on for some time after the boot is done (hence you cant see the physcial volumes until its already tried to activate your VG)
4. kernel loads, ioscan lists all devices as CLAIMED or not.
5. lan driver and network start. You should log into your server asap and do an ioscan -fk to see if all disks are CLAIMED at this point or not. In your example they must have been NO_HW at this point because their VG couldnt be activated
6. Once the box is up fully ioscan must at some stage report your disks as CLAIMED (even if you need to do another full ioscan -f) so that you can manually import/activate them without losing any data.

You need to do tests like this next time you reboot to help narrow it down. You may even find it was a one-off and all reboots fine now.

[FLQ]

James, nothing in syslog.log or OLDsyslog.log... :-((

Very funny problem don't you think...?

I cannot reboot the box now I have to issue a change request before doing so...:-(

I will try to do a couple of reboot and see if I can reproduce the problem

Thanks to all

[Victor_5]

After you run ioscan -fn, do
diskinfo on your specific disks, in your case, it is

diskinfo /dev/rdsk/c20t0d2
diskinfo /dev/rdsk/c27t0d2

To see whether you get the correct output, if "size" and "bytes per sector" say "0", that means you have h/w problem, call HP CE to replace.

[Erik Windischman]

Hey Fracis,

In addition to all the other great suggestions, you may just want to make sure you FC cards are up using fcmsutil.

Get the device names:
ioscan -fnC fc

them run fcmsutil

fcmsutil /dev/"FC device"

You can add "stat" to the end of the command if you want statistics, but if your not an engineer that knows what to look for it probably will do you no good.