Issues with EVA+SecurePath 3.0E for EVA and XP with VxVM/DMP? Anyone running EVA+XP w/ no issues?

[Alzhy]

We've a relatively new (less than a year old) environment consisting of PA-8800 based cell partitionable servers (SuperDome/rp8420), EVA 5000 arrays and a brand new XP12000. We're running HP-UX 11i v1.0 on these environments. We use SecurePath 3.0E for path protection on the EVA LUNs and VxVM DMP for the XP LUNs (with VxVM ASL12K). The boot disks/OS/swap are still managed by LVM while all others are managed by VxVM (with rootdg on small sized LUNS).

Of late, we are experiencing recurring panics as a result of what HP calls WSIO memory arena corruptions. They have not been able to specifically trace the source of the problem after about close to 3 months of troubleshooting.

Is anyone on this forum running the same configuration as we are -- i.e. EVA5K + XP array on the same server? And are you also experiencing the same problems as we are?

[Duncan Edmonstone]

Nelson,

If your concern is that this might be a SecurePath problem, why not remove SecurePath from the equation. HPUX will talk to an EVA down a single path - why not just zone off one controller of the EVA and un-install SecurePath - sure that leaves you exposed in terms of path failure, but it seems like you have bigger problems at the moment anyway. It would at least allow you to prove if the issue is SP or not.

HTH

Duncan

[Alzhy]

Duncan, thanks.

The reason for this post is alternately to find out if we are the only environment that has this unique environment (HP-UX 11i + EVA5K w/SecurePath + XP12K Array w/VxVM DMP AND using VxVM and ancilliary Veritas tools.

If we are, then we're probably seeing a very rare condition that unfortunately is seenin this hardware/software combination that we're running.

Why do we have both EVAs and XPs both being used on our environments?

We use XP as our Primary storage. For PATH protection and load balancing - we employ VxVM's DMP (hence we also use VxVM as volume manager). We also use a product called Flashsnap which allows us to do mirrors/snapshots so resynchs of split mirrors happen faster as only changed blocks are synched up. We use the EVA storage to mirror (via Flashsnap) our XP production data. For the EVA - we use SecurePath 3.0E for HSX arrays. We do not use SecurePath 3.0E for the XP array since it does not support VxVM - hence we use DMP provided by VxVM since we're fully licensed anyway.

Somehow - the most possible theory is during the mirror/flashsnap process where writes/reads hapen simultaneously on both XP and EVA luns - that WSIO memory arena corruption happens. This corruption sometimes results in a panic.

So.. are there indeed sites out there that use XP and EVA storage at the same time on HP-UX 11i v1.0 environments?

[generic_1]

First thing I would do get HP, Flahpoint, perhaps Veritas on a bridge line. Your company paid big bucks and that should help cut through finger pointing runaround.

I did a search and noticed there are some mentions of WSIO in the 11.00 patches so it might not hurt to add those enhancemts incase they fixed some little wigit that happens to be yours

You can used PV links built into your OS instead of secure path for the XP and still maintain your production redundancy there. So removing securepath from the picture might not be a bad IDEA in tracking things down, but just because it works without securepath, doesnt nessarily mean the code problem is there :).
Also in terms of performance if you layed your disks out well across your controllers, use small lun sizes, did LVM stripes when creating your filesystems or raw volumes, and did a good job laying out your san, Securepath becomes kinda a expensive bandaid :).

On the EVA5000, it has an active/passive controllers so so securepath may be beneficial in redundancy situations, but since its your backup you probably have a little more tollerance to loosing that momentarily vs Production hopefully :).

Also if you have a little room someonewhere, setup a test environemnt and start replicating the situation, and playing with the variables so you can track it down quicker.

Also HP has labs too, So they may be able to help since they have the hardware, and this sounds like a serious problem. Good thing is you have all HP hardware basically it sounds ;).

Make sure you HBA drivers and disk drivers are up to date too.

Hope this helps. I do run EVA5k and XP1024, but not in a snapclone setup as you have. We have BCVs on XP or do LVM mirrors, and RMAN backups on some Oracle stuff.

[generic_1]

Is Flashpoint resold by HP? Im thinking StorageTek, may already sell this product is that correct?

[Alzhy]

It is actually called FlashSnap option of VxVM. It's formerly called Fast Mirror Resync (FMR). And yes it is also available from HP along with the integrated VxVM 3.5 on HP-UX 11.i+

HP has comitted to duplicating our environment to see if they can reproduce the problem and hopefully we'll hear from them soon. I am just simply doubtful if within HP and people on this forum - such a similar environment exists - and that is using HSX based arrays alongside XP arrays plus VxVM.

[generic_1]

Doesnt the XP 1200 offera solution to virtualize your storage so you can basically hook your EVA5000 up to it. I thought that was part of the 1200's big selling point other than IO.

extra veritas stuff has more overhead on system than plain old LVM :).

What are you snapcloning, database, filesytem, everything?

[generic_1]

Are you running all the latest Veritas patches, HBA, and Disk as well?

Vertias has had isssues in the past :).

[Alzhy]

Unfortunately the EVA5K cannot yet be hooked up to the XP12K and be virtualized.

Why did we end up with this architecture -- VxVM (and host-based mirrors/backups via FlashSnap) + XP12K (as prod storage) + EVA5K as backup/mirror/DSS/etc.? Becuase we no longer trust monolithic array solutions.. EVA5K's as well as other arrays can fail. If you have your prod as well as your backup/snap/clone mirrors on the same array and it goes down hard - then guess what?

Sure the XP12K has in-array backup solutions (i.e. BusinessCopy or Hitachi HORCM).We do not employ it as we want to still utilize our EVA5Ks for that purpose so what runs on the XP12K are purely production data, so whatever we don with backups, DSS, tests, etc. do not impact the XP/Prod environment.

How do we do it? We use VxVM's FlashSnap technology to mirror copies of our Multi-TB DB instance XP storage to our EVA5K arrays. Flashsnap is intelligent enough to synch only changed blocks so the only pain we have to undergo is the initial synch-up. Subsequent reynchs happen fast (depending on what changed of course) since we're doing resynchs at least once every 24 hours. Each production set can have up to 4 sets -- 2 for backups (that are pushed to tape ona dailybasis) and 2 sets for DSS or development. Since the mirror/backup sets are on a different array (the EVAs).. activities done onthose sets do not impact the XP12K and in an outage of the XP12K, we can quickly use the EVA5K mirrorsets.

Ever since implementing this architecture.. our backups now happen consistently and happen on-time and our weekly DB refreshes (for DSS and test/development) happens faster and are consistent. When we were using the EVA5Ks using BC Snapclones -- our backups and DSS/DB-refresh activities severly impacts our production and performance was unpredictable.

[Dave La Mar]

Nelson -
We recently purchased (April) an EVA5000. This is currently fibre attached to RP7400(s) as well as an XP512. Also running SecurePath A.3.0E.
To date, we have not seen what you describe.
Not likely we will either, as we are near total migration from the XP.

Sorry I cannot be helpful; I am simply addressing your "like environment" inquiry.

Best of luck on this. I'll be one, of probably many, interested in the results.

Regards,
dl