topic Re: reboot caused vg to disappear in Operating System - HP-UX

reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 13:33:11 GMT

Hi all, I had a little problem with one of my L-Class server.

Here is the desc.

L-2000 2 cpus
1,5 GB RAM
2 FC hooked up to a brocade
1 LUN defined for that machine with 2 paths (c20t0d2,c27t0d2) holding SAP DB on a VA7100.

I needed to reboot the box and I did not make any changes to that box concerning the VA but when it came back up, I had lost the vg pointing to the VA resulting in a down DB... :-((

I did try to mount the FS on VA by hand but cannot query physical device.
I did a vgexport -v -p -s -m /tmp/that_va.map that_va = success

I tried then to vgexport for real = success.

I then tried to vgimport that_va using that_va.map but it could not find anything on the disk devices.

At that point I had to do a vgcfgrestore to make it work and it did, I was able to mount the FS.

My question would be simple...How come I lost that_va when I rebooted?

What might cause that?

Where should I look for clues of what happened?

TIA

Francis

Re: reboot caused vg to disappear

Martin Johnson — Thu, 27 Jun 2002 13:37:35 GMT

do "ioscan -fn" and see if everything shows up and is CLAIMED. If there are things missing check the brocade switch.

HTH
Marty

Re: reboot caused vg to disappear

Stefan Farrelly — Thu, 27 Jun 2002 13:38:55 GMT

If you say you couldnt query the physical device then its changed. This is very unlikely to have happened on the HP, more likely on the remote disk device or whatever controls the brocade switch.

So your old LUN with c20t0d2 and c27t0d2 now have different device names in your recovered VG ?

Re: reboot caused vg to disappear

PIYUSH D. PATEL — Thu, 27 Jun 2002 13:42:14 GMT

Hi,

Check

#strings /etc/lvmtab and see the contents and check for c20t0d2,c27t0d2

Do an ioscan and then check out whther these device files are there and claimed.

Piyush

Re: reboot caused vg to disappear

George Abraham_1 — Thu, 27 Jun 2002 13:54:12 GMT

hai

I belive you need some patches... have the latest Hardware extension patches?

regards

Re: reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 13:54:28 GMT

I did an ioscan and devices were showing up as CLAIMED

The vg is still on the same devices. c20t0d2 c27t0d2

But I could not query the devices when I did a vgimport
even a vgscan did not find the headers....

Thanks

Re: reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 14:04:53 GMT

Unless it fixes that exact problem and it is stated in the text of the patch, our policy is to not apply patch bundles without first testing it.

That said, I do not have the latest HW bundle installed

Re: reboot caused vg to disappear

PIYUSH D. PATEL — Thu, 27 Jun 2002 14:05:31 GMT

Hi,

mv /etc/lvmtab /etc/lvmtab.old
vgscan -a

Was your system running properly before and was this the fisrt time you had this problem ??

You can also load the latest patch bundle on your server to prevent such problems in the future.

Piyush

Re: reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 14:07:51 GMT

The system was working fine before I had to reboot.

I did mv /etc/lvmtab /etc/lvmtab.old

I did vgscan but it did not return the PV that belonged to that_va

Re: reboot caused vg to disappear

Stefan Farrelly — Thu, 27 Jun 2002 14:09:56 GMT

If your'e still on the same phyiscal devices and these same devices werent visible when you rebooted then only 2 possibilities;

1. Had the server finished rebooting when you logged into it and checked if the VG's were mounted ? Next time you boot your server log in from another server as soon as you can ping it and do an ioscan -fkCdisk and see if your 2 physical devices are CLAIMED or not, this will tell you if they are being slow in responding to the HP and point to the fibre switch or disk unit having the problem.

2. It could be a problem with the fibre connection to the disk unit. Either the fibre connections arent good, or there are too many devices on the fibre switch (causing a delay in the HP seeing the drives) or a problem on the disk unit. Any of these will cause your symptoms - initially the physical disks arent available but a little while later you can use them.

If you do a full ioscan -fkCdisk how long does it take ? This is a good indiciation of how fast your fibre connections are - it should come back withing 40 secs max. If it takes a lot longer then you have a fibre connection problem.

Re: reboot caused vg to disappear

James R. Ferguson — Thu, 27 Jun 2002 14:10:17 GMT

Hi Francis:

This would appear to have been a transient disconnection. I presume that your filesystem is intact, since you don't indicate otherwise. Assuming no corruption of the LVM headers, then it is odd that you couldn't query them with 'vgimport'. 'vgcfgrestore' would only replace LVM information.

It would be (and would have been) interesting to query the physical volume with 'diskinfo':

# diskinfo -v /dev/rdsk/cXtYdZ

Regards!

...JRF...

Re: reboot caused vg to disappear

Martin Johnson — Thu, 27 Jun 2002 14:14:29 GMT

Do you have STM installed on the system? You could run it and see if it logged any errors.

HTH
Marty

Re: reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 14:21:24 GMT

Stefan, a full ioscan did not take longer than usual and devices were CLAIMED. I also did a rmsf -H 0/4 0/7 to remove special files then a insf -e to recreate them. It behaved as expected.

James, a diskinfo /dev/[r]dsk/dev/c20t0d2 returned ONLINE status without errors also fcmsutil /dev/td[01], returned also an ONLINE status and no errors.

Thanks for prompt relies

Re: reboot caused vg to disappear

James R. Ferguson — Thu, 27 Jun 2002 14:32:15 GMT

Hi (again) Francis:

By the way, did /var/adm/syslog/syslog.log (or /var/adm/syslog/OLDsyslog.log) yield any insight?

Regards!

...JRF...

Re: reboot caused vg to disappear

Stefan Farrelly — Thu, 27 Jun 2002 14:39:32 GMT

This is going to be hard to diagnose now without another reboot, or a few reboots.

What I think happens is this;

1. HP reboots.
2. Fibre card activates (link light comes on) once the HP is loading vmunix (not before) as it needs to load the fibre driver in the kernel first.
3. You should watch the link light on your fibre card while it is rebooting to note when it comes on. It may not be coming on for some time after the boot is done (hence you cant see the physcial volumes until its already tried to activate your VG)
4. kernel loads, ioscan lists all devices as CLAIMED or not.
5. lan driver and network start. You should log into your server asap and do an ioscan -fk to see if all disks are CLAIMED at this point or not. In your example they must have been NO_HW at this point because their VG couldnt be activated
6. Once the box is up fully ioscan must at some stage report your disks as CLAIMED (even if you need to do another full ioscan -f) so that you can manually import/activate them without losing any data.

You need to do tests like this next time you reboot to help narrow it down. You may even find it was a one-off and all reboots fine now.

Re: reboot caused vg to disappear

FLQ — Thu, 27 Jun 2002 14:48:34 GMT

James, nothing in syslog.log or OLDsyslog.log... :-((

Very funny problem don't you think...?

I cannot reboot the box now I have to issue a change request before doing so...:-(

I will try to do a couple of reboot and see if I can reproduce the problem

Thanks to all

Re: reboot caused vg to disappear

Victor_5 — Thu, 27 Jun 2002 16:42:34 GMT

After you run ioscan -fn, do
diskinfo on your specific disks, in your case, it is

diskinfo /dev/rdsk/c20t0d2
diskinfo /dev/rdsk/c27t0d2

To see whether you get the correct output, if "size" and "bytes per sector" say "0", that means you have h/w problem, call HP CE to replace.

Re: reboot caused vg to disappear

Erik Windischman — Fri, 28 Jun 2002 13:19:06 GMT

Hey Fracis,

In addition to all the other great suggestions, you may just want to make sure you FC cards are up using fcmsutil.

Get the device names:
ioscan -fnC fc

them run fcmsutil

fcmsutil /dev/"FC device"

You can add "stat" to the end of the command if you want statistics, but if your not an engineer that knows what to look for it probably will do you no good.