Problem restoring with Ignite on system attached to SAN.

[Mike Keys]

After querying various groups and our 3rd party OS support group, I felt confident in the steps needed to Ignite (make_tape_recovery) our N4000 systems and use this tape to restore to larger hard drives.

I ran the recovery in preview mode and edited the "system_cfg" file, modifying the 'MAX PE per PV' and the 'PE size' parameters to take advantage of the larger disks (18GB to 36GB). These were sized as suggested by another ITRC forum member. MAX PE per PV = 10000 and PE Size = 16MB. According to this individual, this would allow for upgrades to 160MB disks in the future without having to change the group parameters.

I encountered one problem after another. The N4000 is attached to a SAN where our database resides. My assumption with the recovery tape (and I assumed wrong) is that this tape would specific to a particular system (i.e. know about the devices, paths, etc.). Therefore, changing a couple of sizing parameters and resizing /stand should have had little effect.

However, once the system was restored and was being brought back up, I saw many errors related to the lvol's/vg's associated with the SAN. Apparently, these weren't found and reported FAILURE. The system then rebooted again and when I was able to get into the console, I discovered that 'lvmtab' was hosed. None of the SAN filesystems were there and we ran out of time to troubleshoot and fix.

So, all of you HP guru's, how is one supposed to make a recovery tape of a system that is attached to a SAN. If this is any indication of the power of the Ignite recovery feature, then it has failed miserably.

We have learned a hard lesson that none of our systems that we run 'make_tape_recovery' weekly on can be guaranteed to be restored to a pre-crash condition.
Mike Keys

[Simon Hargrave]

Were you restoring the tape to the server on which it was created, or onto a different server attached to the same san?

If the latter then it's likely you'll need to configure the zoning/lun masking on your san to ensure your restored server has access to the same luns as the server you backed up.

Once this is all correct have you tried using vgscan to re-read the LVM information from the visible LUNs?

[RAC_1]

What was the command that you used?? Was your database residing on vg00?? If not did you include the vg (residing database) in the recovery backup??

-x inc_entire=vgxx

The ignite is really for system recover-root disk recover. For data other than this, you should be using other backups-OB, tar etc.

Anil

[Mike Keys]

I ran `make_tape_recovery -i` (interactive mode) and did select vg00 and it showed "inc_entire" in the appropriate column. The restore was done to the exact same system. The only difference is that I changed the MAX PE per PV and the PE Size parameters in the system_cfg file as we moved from 18GB to 36GB OS drives.

We are using 6.1.44 version of Ignite. When I ran the strings /etc/lvmtab none of the SAN vg's or lvol's were there.

I did not run vgscan as I did not know to do this.

[Florian Heigl (new acc)]

look if You got some ct_query failed messages in dmesg or syslog, these would indicate a zoning problem.

a ignite tape does contain the full LVM information, so something must've gone wrong. :)

[Patrick Wallek · Accepted solution]

The easy way to get your VGs back in this case:

Prior to running an ignite tape:

vgexport -p -m /home/root/vg??.map -s vg??

Where vg?? are your VGs. The '-p' switch is preview mode so just the map file will be generated.

After restoring from the ignite tape, reimport your VGs with:

vgimport -s -m /home/root/vg??.map vg??

If you /dev/vg?? structures are not there they may have to be recreated.

I have had cases where all of my VGs came back as they were before the Ignite restore and I have cases where I had to re-import the VGs.

Changing the VG00 parameters would have had no effect on the other VGs. And I'm not sure why the 10000 didn't work for you. At times you just have to experiment until you find a value that works. With the 4340 value you mention in your other thread and 16 MB PE size, you can go up to a 69,440 MB disk.

[Mike Keys]

Well I couldn't tell you what messages we got, because we had to place the original disks back in the system. It is not a zoning issue.

When we first did the restore, the Ignite program runs an ioscan. On our server it hit the fibre cards running to the switches. We use EMC PowerPath with our CX500. The program would grab two device paths at a time and then report back that a disk with a 0 size was found. Here is an example of those device paths:

0/10/0/0.97.4.19.0.0.0 0/10/0/0.97.4.19.0.0.1
1/8/0/0.97.5.19.0.1.0

We have many of these paths because of the way the switches are zoned. There was a delay of 15 - 20 seconds between the time the program grabbed two device path before grabbing another 2. So, the entire scan process took about 1/2 hour.

Once into interactive mode (we only put in 1 OS drive), we saw that the VG group properties were wrong (i.e. the disk size being reported was not 34720MB but approx 62GB because of the disks being reported attached to the SAN!)

Plus, there was an error being reported with one of these fibre paths. Ignite would not continue until the problem was corrected.

Our HP CE decicded to unplug the fibre cards and run the restore again. This time went right into the Ignite program. Made changes and started restore. At this point he plugged the connectors to the fibre cards back in.

So, must the scan run during the 1st part of the restore in order to see these paths later? I would think so if the information was contained in the lvmtab and lvmconf files/directories.

[generic_1]

Unless you are restoring data to your san, Unplug your fibers untill the install is done. Depending how much storage you have it can cause issues for ignite. Install your fibers when you are done, then reboot.
The basic issue is the mini kernel does not deal well with too many devices unless they have found a way around that.
On your making of the tape use the include option for entire vg00 and you should be cool. Make sure you take regular good backups of your application data on your other diks/volumes before a major project like this if you intend on keeping the data.

Best of Luck