EMC BCV refreshes on Itanium server issue....

[Rita C Workman]

I've been using EMC technology to create a Business Contiuance Volume (BCV) for years. Up till recently running this mirror sync/split process called BCV (Timefinder) on our 11i PARisc server.
But I recently moved this process over to an Itanium (11.31) server.
All seemed to working fine.....Until -

Discovered that if I made a change in sizing at the LVM layer on the primary side, the change is not being reflected on the BCV side.
Now, if I add a new lvol I expect to vgexport/vgimport and re-set up things on the BCV side. But for simple lvol increases, generally I don't have to do anything. The disks get sync'd and all is there.

But since going to Itanium, if I increase the size of an lvol, even with existing disk in the volume group, it is not reflected on the BCV side. I can reset up those process on the BCV side and vgexport/vgimport everything, but the new lvol sizing will NOT be reflected.

To get around I'm fixing the BCV side manually for the lvol.

Does anyone know of any HPUX patch to address this situation?

....and yes, I have also posted this on EMC's 'Forums' to see if anyone there has seen this.

Many thanks,
Rita

[kevin_m]

Did you by chance try an establish with the -full option?

Kevin

[Bill Costigan]

Are you saying that even a new vgimport does not show the correct lvol size?

If so, it would seem that the changes are not being written to the disk. By chance have you tested to see if deactivating the VG on the source server, followed by the BCV sync/split would cause the changes to be seen?

[Rita C Workman]

Kevin:
Yes I did both a full establish on the disks involved associated to the change, and a full for the entire volume group.

Bill:
Exactly that. Doing a new mapfile and using that to import on the BCV side shows absolutely no change.

Yesterday -
Ran full re-establish on disks.
Manually increased the lvol in question to reflect the new disk and the new disk.
Ran vgcfgbackup on BCV side
Ran fsck -F vxfs -y -o full /dev/vg/lvol
>>>and it finally ran.
Mounted the file system !!

>> I thought things were looking up <<
>> Wrong <<

The automated script ran last night to resync the BCV disks and mount on the BCV server.
Script ran doing incremental refresh (normal), but the lvol in question now appears to be back to the old state.
lvol does not show manual increase made yesterday.
File system will not mount.
Same errors as before.......

/rcw

[Rita C Workman]

Bill:

I can not de-activate the volume group on the production side. It is production!

/rcw

[Olivier Masse]

Rita,

Did you check if you're using LVM 2.x on these volumes? If that is the case, one could assume that maybe, just maybe, that used to work with 1.0 volumes only. But you'll need to read the white paper or open a call to confirm this. I scanned through it for 30 secs and didn't find anything describing your problem: http://docs.hp.com/en/lvm-v2/L2_whitepaper_8.pdf

Besides this, someone suggested deactivating the VG but I'm not sure this could help. But if you want to try something, instead of deactivating you could try to quiesce the VG for a few seconds, the time to start your business copy (it's a new option in 11.31, check the vgchange manpage or the whitepaper on docs.hp.com). It mentions in the man page that "The physical volumes in the volume group appear as though the logical volumes were closed and the volume group was deactivated." which should give you a similar effect without impacting the production side. If you don't have a test server to play with, I can only suggest that you be careful. :)

In all cases I'm sure EMC has top people who could help you with this.

Good luck

[Rita C Workman]

Hi Oliviere,

Did you check if you're using LVM 2.x on these volumes?
>>The BCV side reflects 1.0. So it does not appear to be a conflict with new LVM vs ols LVM versions.
I downloaded the document, but so far I'm not seeing anything either.

The primary side, I can't touch that without hearing about it (probably loudly...). And that side is running 11.23 PARisc. It's the BCV side running 11.31Itanium.
When the BCV (11.31) is refreshed the primary side is set into hotbackup mode and done during early am (quiet time) hours. Since it's the BCV getting sync'd I can't see where quiescing them would help.
Let me clarify for folks...
The script that does the sync does the following:
ON BCV SIDE:
It umounts all BCV mounts
It deactivates the volume group
It then re-sync's all BCV disks back to primary.
It splits off BCV mirror copy disk
It double checks to ensure all background processes are done.
It reactivates the volume group
It does fsck on every logical volume
It remounts the file system

Like I said...if I manually go in and fix the volume group & resync that logical volumes disks (no other lvol shares any of these disks!), then fsck lvol manually - I can then mount it.
But at next automated running of the script I'm right back where I was again.

Going to have to open a call ticket with EMC, cause so far even the folks on EMC Support Forums seem stumped.

/rcw

[Steven E. Protter]

Rita,

So nice to talk to you. I warn I have no insight on this problem, but will tell you how we handle it and maybe it can make your life easier.

We never resize any LUNS in our EMC environment.

If server mngp01 needs a 300 GB LUN, it is presented via EMC as 10 30 GB disks. We create a volume group, extend it to include the disks, set up a logical volume and use newfs to set up the file system.

Lets say this 300 GB mount point needs to expand to 330 GB. Then we present another 30 GB LUN, vgextend this new disk, lvextend and then fsadm (Online JFS) to extend the filesystem with no down time.

I first had the exact same problem you are having with another storage vendor. HP promised me a patch to deal with this, and years passed without the patch. I like the current model we use here at Dana much better.

Reize just doesn't work reliably.

SEP

[kevin_m]

It's a long shot, but how about doing a 'symmir cancel' for the STD/BCV pair then running the establish again? The idea being that Timefinder retains the original size of the standard and will only sync that number of extents. Just a thought anyway.

[Rita C Workman]

Well Hi Steven,

So, you loving Toledo (such a hotspot...teehee)!

I would go along with that approach if I was talking Windows. But I do not follow that logic with HPUX.

And what is "annoying" me, although that is probably not the best word, is that I have been doing this for many years and never ran into this until I decided to move all the administrative functions (Data Protector,OEM,Documentation_webpage....and BCV refreshes) to the Itanium platform.
It works fine using your approach...create new lvol with new f/s and it is ok.

But just increase the lvol (even with spare disk already in the volume group) and it is not handling the change.

AND....if I manually adjust the BCV side on a space increase to match the primary side and get it all up and mounted on the BCV Itanium server. The flipping thing brain-farts at the next running of the automated BCV refresh script. And I'm back to where I was. Exactly as before. With the volume group (which I changed and saved) is back to the old (pre-size increase) settings on the BCV side.

I now have call tickets opened with EMC & HP, but so far they haven't called.

Just sign me....
Waving to ya from WV!
Rita

[Steven E. Protter]

Rita,

I understand your frustration.

Toledo is great. No traffic. I can get anywhere in 15 minutes. if I had a car. which I don't.

I don't consider our approach windows centric. LUN resize has proven to be difficult with HP-UX, and here at Dana they have a way around it.

You are just now feeling the lvol reize pain that I felt years ago in Chicago.

Bottom Line: There isn't going to be a patch and a new approach will probably be needed.

SEP

[Duncan Edmonstone]

SEP I think you are getting confused here between lvol resizes and LUn resizes...

as well as I can understand here Rita is talking about lvol resizing only - she isn't changing the size of any of the LUNs presented to her system(s), she is simply adding extents to an existing logical volume and expecting that size to be shown when she activates the VG and mounts that LV on the 11.31 system - pretty normal practice to be honest.

If the lvol really isn't showing its new size it has to be for one of 2 reasons:

1. when the disks are synced, for some reason the sync isn't working correctly

2. when the vg is activated the data read in from the VGRA headers isn't being interpreted correctly on the 11.31 system for some reason.

Can we go back to some basics here Rita and try the following:

1. on the 11.23 system run a lvdisplay of the effected LV - note the number of physical extents.

2. vgexport the VG on the 11.31 system

3. resync and split your BCVs

4. vgimport the VG on the 11.31 system and activate

5. before attempting anything else run a lvdisplay of the lvol in question - does it match what you saw on the 11.23 system or look different - can you post the differences if there are any?

HTH

Duncan

[Rita C Workman]

Kevin:
Tried the cancel option and did a full establish on the affected disks. No change.

Steven:
I'm not changing the lun, I'm just increasing one lvol with disk already in the volume group. This has never been an issue for me, not even with the BCV side. Not until now that I've put the BCV side on an Itanium server. If I add a new lvol and give it disk, it's fine. It's only when I increase existing diskspace to an lvol that the BCV is having issues.

Duncan:
Good suggestion and I have it a try on the BCV side, but it just plain is not taking. As soon I run the import I'm back to where I was before. On the BCV side, the lvol size increase is NOT reflected.

I've confirmed every disk and it's association to every BCV (mirror). I've exported the vg on the BCV side and re-synced (full) for every disk in the whole volume group (not just the lvol). And vgimported using the -f option to pick up only the BCV disk (confirmed those numbers too!). And still it isn't working.

But if I take a hammer to it. Manually fix the lvol on the BCV side and resync. Then on the BCV side I can fsck the lvol and mount the silly f/s.

But come next refresh cycle, it will deactivate the volume and when it re-activates it after, the lvol is back to the old sizing.

arghghgh!!

I have a headache...and so far no resolution from anyone.

/rcw

[Duncan Edmonstone]

Rita,

So can you confirm that the sync _is_ working correctly by:

- create some new files in the filesystem on the 11.23 system
- ensuring everything is flushed to disk (sync;sync;sync)
- run through the "hack" you have to do to get the filesystem/lvol sized correctly on the 11.31 system
- do you see the new files?

That will tell us the EMC system is syncing disk correctly... assuming it is, you need to get the support organisation to step up - in my experience if you haven't had resolution from a call after serveral days, you should be asking for it to be escalated to L3 support.

HTH

Duncan

[Rita C Workman]

Duncan,

Yes I have confirmed it is syncing properly.
I have support calls into EMC & HP and am working thru the usual "it's their software or your script" statements (that I expected).

I've managed to get HP escalated to next level. EMC is reviewing all the info I passed them per their requests, so I'm hoping they can contribute something more.

I'm going to leave this thread open....if I get a final resolution I'll post it for anyone else who might ever hit this. For now, like I said I can fix it each time manually and get the mount back up - but I head out on vacation in 10 days....and this has to be fixed before I leave.

Thanks,
Rita

[Bill Costigan]

Is there any chance that the source of one of the mirrors is not really the actual LUN used in the source VG but a point-in-time copy of the LUN?

Because the changes you make in the VG on the 11.31 server is being over-written when you re-sync then clearly the resync is working and the data must be coming from the source LUN. But this data is old so either the 11.23 server hasn't written the info about the lvol being increased (which seems unlikely) or the source is a different LUN than is being used for the VG on the 11.23 server.

[Rita C Workman]

Well, after doing a whole lot of testing with HP folks, it appeared that maybe just maybe the sync wasn't working right.

In the end to fix this, for anyone who uses EMC software and BCV's, I took the BCV side for the volume groups down. Exported the vg; and on every one of the associated BCV disks to the affected vg(s) I did a pvcreate -f and rebuilt everything like it was the first time.
Did a full establish on the disk; reimported the vg; reactivated everything ran fsck on the lvols and was able to bring it all up.

Ran my automated script that handles bringing things down-resync-bring it all back up a couple times to ensure everything was holding....and it finally did.

Never had this kind of trouble before when I simply did lvextend in a BCV environment prior to moving my BCV's side to the Itanium platform.

But I'm working........

Thanks to all for their thoughts and suggestions.

Rgrds,
Rita