EVA 6000 XCS 6110 Performance Issues

[stoney88888]

We recently did a controller upgrade to 6110 from 5110 after a lid failure. Since the upgrade we have had slowness issues with one server in particular that handles our backups. We do disk backups to a LUN presented to this server which used to take 30 minutes now takes over 2 hours. Prior to the upgrade we upgraded the firmware on all of our HBAs, Fabric Switches, plus drivers and latest MPIO. This is running Win2k3 with MiniStor port drivers. Ran EVA Perf with no real problems that stand out. We are running Emulex LPe1150 HP cards. Has anyone experienced this issue? Thanks John

[Rob Leadbeater]

Hi John,

Are all of the other servers running the same software and drivers ?

Due to the number of changes you've made, it's difficult to see how you've come to the conclusion that its the XCS code that's at fault...

Cheers,

Rob

[stoney88888]

We standardized on the same drivers for all LEe1150e HBA drivers, firmware, boot bios, and MPIO. I guess I am grasping at straws since it didn't happen until after the XCS code load.

[McCready]

Just a few of the "standard troubleshooting" tips (sorry if you have already done these):
- latest in all does not mean that the specific mix is supported; if possible, validate that what you have is supposed to work with each other.
- Investiate your switch firmware and port settings; we just did an upgrade what we thought was the latest and greatest Cisco firmware a month old, just to see a bugfix release come out a week later. Also, your switch port settings may have changed during your upgrade, depending on how they were saved (or not) prior to that.
- You will probably want to zero any counters you can, and track any errors on any ports, hba's, event logs, etc, that show up as part of your backup.
- Check your MPIO settings - you may be better off turing off any load balancing there, or selecting specific paths to use to isolate a path problem if possible.
- Make sure your lun is using the same level of raid as it was before.
- Lastly, make sure your backup software has no issues doing I/O to the cards or other parts of the I/O setup in question.

Just a few ideas...

[stoney88888]

There is a bug in the 6110 code that is widely unpublished. The problem is that the Read Cache can become disabled after the code load. Basically the prefetch is set to all zeros so when doing a sequential write the controllers are not prefetching and storing the information in read cache. Thus slowing down the performance tremendously. So right now it is a bug and there may be a remote possiblility that resynching the controllers may fix the issue. Unfortunatley this will require us to take down most of our servers since we are heavy boot to SAN and have some pretty unforgiving proprietary dbs that will corrupt if not taken down. Stay Tuned!

[ALCS]

hi,

How was your problem solved?
I am now reluctant to upgrade to 6.11
unless the problem can be resolved by a reboot.

thank you for updating

farid

[stoney88888]

I have mixed feelings for you on upgrading. We are only 1 of about 10 sites throughout the US having this problem. It only seems to affect when doing a sequential write to disk. This is how our backup works so this is how we noticed it. 6.2.2.0 is supposedly coming out mid march which they have promised will fix this issue. The 6.1.1.0 code other than our issue is pretty solid. The reason that we went with 6.1.1.0 was because we had a LID failure that locked up both controllers. So they fixed the LID failure issue but introduced this issue and really because the code will work on the controllers for the EVA 4X00, 6X00, and the 8X00. The 8X00 has HSV210 controllers which are different from the 4k and 6k controllers and have much more cache memory in them so it really only affects the 4k and 6k. I would talk with your FE on the upgrade to see what s/he thinks. Regards, John

[MWard]

I am having the same problem since upgrading to 6110. After performing some tests it appears to only be really affecting our Windows servers, but the performance is severely degraded. Does anyone know if HP is planning to fix this bug in the near future? What has been the success rate of a controller resync fixing the issue?

[Tom O'Toole]

A resync should not in theory, cause anything user visible to happen. The paths should failover and fail back in like 45 seconds. And in practice mostly this has worked for us with Openvms and even aix (can't say for MS) , but I think we have had a few problems. Of course there is the 45 second hang that users experience. Then, like you say, some databases can freak out over 45 seconds of no I/O. It's amazing in this day and age that these expensive, supposedly 'enterprise-ready' products to which we entrust our data are, in fact, quite fragile.

Someone will probably chime in, "well you have to set resync_flap_avoidance_datacheck_mutex to 50 and reboot", but come-on , I feel this stuff should have better recovery out of the box.

[stoney88888]

Unfortunately are database structure is written in a proprietary format for an application written in MUMPS. A 45sec hit would indeed kill and possibly corrupt the databases. SQL and Oracle may be able to handle it but not our current dbs. We are also very heavy boot to SAN so this will also cause issues.

As far as HPs fixes it is supposed to come out mid march version 6.2.2.0. Hopefully it comes out on time.