Can't succesfully mirror.

[Joe Short]

I have an rp7420 running HP-UX 11.11.
It is currenlty attached to a pair of SANs.
One SAN has a CX500 array that is currently in use. The other SAN has a CX3-40 that we are trying to migrate to. I have created LUNs on the CX3-40 that are the same size as those currently in use. I have added those new LUNs to my VGs. However, whenever I try to mirror onto the new array using MirrorDisk/UX, it runs really slowly and then fails. Whe I attempt to continue by using either lvsync, or vgsync it again proceeds slowly and fails with the error message "Could't re-synchronize stale partitons of logical volume: Device offline/Powerfailed"
The array is not giving any errors nor ar the switches. I am under a deadline and need to get this resolved as soon as I can.
Any help will be welcomed.

[Tim Nelson]

Maybe take a step back.

Can you use your new disk for anything ?

Pull the devices out of vg00 and dd some data to the new disk, do they work at all ? Is the performance reasonable ?

[TTr]

"Device offline/Powerfailed" may indicate an issue with the new LUNs. Do you get any errors in the syslog? Monitor the progres of lvsync by using lvdisplay on the lv that is being sync-ed.

[Joe Short]

This is the strange thing. I have about 20 VGs that I need to migrate to the new array.
A few of them went smoothly, I added LUNs from the new array into the VG, mirrored the LV onto it, the reduced the LV off the old array and reduced the old array out of the VG. On those VGs, yes, the performance is fine.

[Tim Nelson]

I would start reviewing array performance while you are mirroring.

There certainly is some timeout or other issue occuring. Could be just about anything. Definately an array / fabric issue though.

[Joe Short]

Well, I did find something, but it seems so minor. The previous admin had set the PV timeout to 180 on the CX500 LUNs. I am resetting it to the same value on the CX3-40 and retrying the vgsync command.

[psreedhar]

Hi,
180 Time out too much. The default value required is 10. Don't set higher timeouts.

run "dmesg" command and see if you find any error messages printed by LVM kernel.

lvsync fail with "device offline" message only when one of the PVs is not available.

Check the /var/adm/syslog/syslog file. I am sure you will find Device offline messages.

[TIMOTHY J. REGAN II]

Are you using EMC Powerpath or HP AutoPath?

[Jim Mallett]

Wow, even Unixgod needs a little help once in a while?

I can't add much for your original issue. My first thought was a bad disk, but if this is happening with multiple mirrors it's not likely. For the mirrors that are failing, are they coming from the same RAID group or physical enclosure?

The only other thing I would say is to try a dd to one of the LUNs that mirroring failed with, see how it goes.

As far as the timeout comment goes, EMC recommends the extended PV and LV timeouts. I emailed you a document on it, probably nothing you didn't already know. They also recommend turning bad block realloc to OFF for your LVMs, and letting the array to the dirty work.

Your humble student....
Jim

[Joe Short]

So here is another symptom. I was told that whenever there is a disk related issue on one (of 3)HP-UX server on the CX500, all 3 of the servers are negatively impacted. Once an internal drive failed on the non production server, and ALL IO traffic on all 3 servers slowed to a crawl. However, other servers (non HP-UX) are not impacted. All lvols are set for strict allocation policy, and bad block is on. I am not sure why the strict policy was set on these lvols since they are on arrays and not normally mirrored.

[TIMOTHY J. REGAN II]

Joe I would suggest that on this CX3?? array that you have a Flare code no later than level 26 and the HP-UX hosts are defined as "Clariion Open". If you have multiple fiber paths pointing to the LUNs; i.e. cross bus mirroring then set the failover type for these LUNs to active/active.

You must also turn off this bad block relocation as the array will handle this for you auto-magically. ;-)

I asked earlier if load balancing software if any was used that could also be coming into the picture or not.

Hope things stabilize soon.