HELP! XCS 11001100 post upgrade latency issues.

[webexec]

Hello. Last night I completed an online upgrade to 11001100 from 09534000. This is on a EVA 6400. Previously I had posted on this forum about an upgrade to CV from 9.3 to 10.2 in preparation for this. This was successful by simply uninstalling 9.3 and installing 10.2. That post is here.

I digress..

This EVA is mainly used for VMware (esxi 4.1 build 1050704) data stores, since I am 98% virtualized its pretty much everything. After the upgrade I am seeing a major increase in write latency across all vdisks. Read latency appears somewhat elevated but write latency is off the charts. I have poured through best practice docs, my configurations in ESXI, EVAperf, and anything else I can google trying to fix but have come up empty.  This is what I see after upgrade:

Here is everything I have changed so far in order.

1. Upgrade CV to 10.2
2. I/O reduced by shutting down all VMs.
3. Upgrade XCS to 11001100 FW
4. P6000 EVA VAAI driver installed on 4.1 hosts immediatly after, hosts restarted.
5.   **Latency noticed after VMs powered on, hosts are boot from SAN.
6. Changed psp to RR for datastores on a hosts where it was missed.
7. Changed psp to RR for RDMs where it was missed.
8. Verified ALL data stores and RDMs are VMW_SATP_ALUA and psp set to Round Robin.
9. Changed Disk.DiskMaxIOSize setting to 128 from default 32787… no noticeable change.

Plan on calling support but figured I would post here as well. I'd appreciate any help anyone can give. At this point I would even consider downgrading back to 09534000. However I dont think that is possible. 

Some things I was considering were: removing the VAAI driver from the hosts (although I cannot seem to find how to do that), upgrade to 11200000 FW in hopes that ‘fixes’ it.

[Marc0]

Has anything changed regarding Cache Policy or Write cache setting ?

Here i got:

Cache Policies - Read cache - On (Mirror cache: Mirrored)

Write cache - Requested - Write-back

Also, i know that if the SAN/Fabric/Paths would be an issue, then Read and Write would have problems, but did you check your fabric for any events, errors after the update ?

And what latencies do you see under "System Options - View management paths" ?

Really interested in this 'cause we have to update some evas from 9534000 too but we will take one more step and update to 10001000 first.

[Torsten.]

>> Really interested in this 'cause we have to update some evas from 9534000 too but we will take one more step and update to 10001000 first.

Yes, this is required!

Finally go to XCS 11200000, but update command view to 10.3 first!

[Jason_L]

I would check the preferred controller settings in commandview.  This to make sure that all the VM's are not accessing LUNS all on the same owning controller. 

Can you provide a snapshot of the LUNs and the current managing controller?

If you are set to "no preference" there is a chance all LUNS are "owned" by the same controller. 

-Jason