HP 3PAR 7400 poor performane / designing question

[michtsch]

Hello Guys

I'm a system engineer in my company. We have a strange behaviour with our 3 PAR 7400. So we think it's a strange behaviour :-).

Either, we weren't able to help designing an implementing the system. The system runs about 1 year and the performance is poor (in our opinion, very poor). If we backup (Veeam and HP DP) we have an average troughput from about 150 MB's (as well if we do a normal copy Job on an SSD LUN). I have more speed on my USB Disk. It was an external company who set up everything.

So our job is now reverse engineering :-), what a pleasure.

We have virtual servers and normal server's, everythig is running in two new Blade C7000 enclousures with virtual connect. installed by this company as well. Here's what we got:

2 x C7000 Blade Encloursure

Each Enclousure has:
2 x HP VC FlexFabric 10GB/24-Port Module
2 x HP VC 8Gb FC Module

Then we have:
4 x HP SN6000B 16GB FC Switches
1 x Store Once 4500 with upgrade cage (VTL for Backup)

2 x StoreServ 7400c
6 x DriveShelf M6710

Each Shelf has: 8 x 10K 900 GB SAS, 6 x 72K 1TB NL SAS, 4 x 150K 480GB SSD

(Same 3 PAR Hardware again for RemoteCopy)

Everything is connected with FC. So in my opinion this should run llike hell. But it don't...

As I allready told the Backup is slow, a copy Job from a hardware server to an SSD LUN is slow (150 MB's). If I run a checkwith I/O Meter, sometimes I won't come over 35 MB's with 16 worker (single I/O). I know, 3 PAR is designed for random I/O's, but I'm sure there have to be more performance as well in single I/O's. Reading is sometimes very fast. I did a copy joby (150GB, a Veeam Backup) from the 3 PAR SSD LUN to the Hardware Server multiple times, several times a week. Sometimes I was on 1,5 GB's, most of the time it went down to about 350 MB/s. If I check the 3 PAR during test's on cli, it just sleeps...

I know there are several things the company who implementet the 3PAR did not a good Job:

- Wrong SAN zoning (Server and every target in one zone)
- SAN Switch FW in Fabric different
- RemoteCopy Job from SSD to FC
- Ports on 3PAR not configured symmetrically

Other thing I don't understand they did: We have 2 3PAR System's. Every Blade Center is Connected with each 3PAR. But one 3 PAR is only used for Remote Copy. Is this some kind of best practice? (I mean the cabling)

How have you guys set up your 3 PAR environment? What performance should we be able to expect? What results do you have with I/O Meter? Do you have any advices for us?

We allready read so many best practices guides for the 3PAR and we now, there is a lot we should change. But I wan't to know how you guys do it in your IT and if you have any advices for us? Especially with the cabling of the Remote Copy 3PAR and the Blade Center's I don't understand.

Many many thanks in advance for your help.

Cheerio

michtsch

[Dennis Handly]

> If I check the 3PAR during tests on cli, it just sleeps

Are you saying when you use statvlun, you don't see much activity?

[yapkl]

These are complex questions which can't be addressed in a forum. 3PAR best practices document is here: https://www.hpe.com/h20195/v2/GetPDF.aspx/4AA4-4524ENW.pdf

As for your config, based on the drive quantities you gave (24x480G, 48x900G, 36x1TB NL) it's not exactly a small starter config so performance wise should be ok. There are so many considerations:

1. Is this a 7400C 2N or 4N?
2. What is the 3PAR OS version?
3. Are you using any tiering (Adaptive Optimization)?
4. Are the hosts using active-active multipath (MPIO, VMware RR, etc..)

Even on a small minimum config 3PAR with SSD (AFC/AO) I can easily push iometer to 50k IOPS at 100% random, 8k, 70% read workloads and well over 500MB/s for 256k sequential writes.

So like you said, you may have to review every single data path to see where it's slowing you down to 35MB/s.

You mentioned you're using Veeam, if your 3PAR has virtual copy licenses, then you can also do Veeam + 3PAR snapshots integration - that could help a bit. https://helpcenter.veeam.com/docs/backup/vsphere/hp_3par_add.html?ver=95

[michtsch]

Jup, you're right. I did the test again with I/O Meter:

1 Worker
32KB 100% Write
1# of outstanding I/Os

cli% statvlun -ni -hostsum -iter 1
09:19:08 04/03/2017 r/w I/O per second      KBytes per sec    Svt ms     IOSz KB
           Hostname      Cur  Avg  Max    Cur    Avg   Max  Cur  Avg   Cur   Avg Qlen
            server1   t 2564 2564 2564  84022  84022 84022 0.33 0.33  32.8  32.8    1

8 Worker
32KB 100% Write
1# of outstanding I/Os
        
cli% statvlun -ni -hostsum -iter 1
09:20:51 04/03/2017 r/w    I/O per second       KBytes per sec    Svt ms     IOSz KBe
           Hostname       Cur   Avg   Max    Cur    Avg    Max  Cur  Avg   Cur   Avg Qlen
             server1   t 12292 12292 12292 402778 402778 402778 0.47 0.47  32.8  32.8    7
        

[michtsch]

hello yapkl

Many thanks for your answer, now at least I have some performance values I can deal with. 

So regarding your question:

1. Is this a 7400C 2N or 4N?
It is a 7400C 4N

2. What is the 3PAR OS version?
We are currentrly running on OS: 3.2.1 (MR3)

3. Are you using any tiering (Adaptive Optimization)?
Yes, we use AO

4. Are the hosts using active-active multipath (MPIO, VMware RR, etc..)
Yes, the using active/active multipathing (MPIO)

Cheerio

Michtsch

[Sheldon Smith]

As yapkl wrote above, "[t]hese are complex questions which can't be [properly] addressed in a forum". That being written, here are some basic observations and questions:

SAN switches.
Firmware. At the time of this writing, 8.1.0a is recommended. The firmware can be updated on-line.
As far as the SAN zoning goes, if the 16GB switches are running fairly current firmware (7.4.x or later), they have added what is called "Peer Zoning". This is in addition to the traditional single-server zoning that has been used for decades.
With a Peerzone, a 3PAR port's PWWN is designated as "principal" and any hosts' PWWNs are designated as "members". A Peerzone isolates the members from each other and only allow communication to the principal. This is part of the mechanism for the HPE 3PAR Smart SAN software. So if the SAN zones are Peerzones, then they may be fine. If they're not, the zones need to be cleaned up. The zoneshow command will display the entire zoning configuration.

Remote Copy.
How far apart are the two systems? Same data center? Miles? Remote Copy over FC or over IP? Synchronous or asynchronous Groups?
Are both blade enclosures adjacent to one system, or is one enclosure close to each system?

Hosts.
What operating system(s) are used? Hypervisors? Virtual guests? Physical servers? Are any guests using a pass-through mechanism to the systems, or do all guests use virtual disks via the hypervisor hosts?
Do the hypervisors have active-active multipath (MPIO, VMware RR, etc..)? I have seen guests with round robin, but the ESXi hosts are still using the default Most Recent Used pathing not Round Robin.

[yapkl]

If you're doing AO, please review your AO config and make sure you're not doing tiering with NL and using cost mode. That will push a lot of data to NL tiers and when you try to access those, performance will be from your NL tiers (assuming no cache).

https://www.hpe.com/h20195/V2/GetPDF.aspx/4AA4-0867ENW.pdf

[michtsch]

Dear Sheldon Smith

Thanks a lot for your answer!

Here my input:

SAN Switches

So I checked the firmware of the SAN Switches. Basically there are 2 Fabrics with 2 Switches each.
Fabric A: Switch1 = v7.3.1a
Fabric A: Swicht2 =v7.2.1d
Fabric B: Switch1=v7.3.1a
Fabric B: Swtich2= v7.2.1d
The zoning is a "mess". I try to explain how they did the zoning:
F.e. Zone for Server1
Server1 (2members)
Storage A Port1
Storage A Port2
Storage A... until Port8
Storage B Port1 
Storage B.... until Port8
BackupServer (44 Members)

It becomes more and more clearly that the didn't exactly now what they did. The worst thing of all, this is in production. So we won't be able to just do a new zoning on the fly. Storage A is connected to Fabric A + B and Storage B is connected to Fabric A + B. (Just for the notes, Storage A is for me one 3PAR 7400c with 4 Nodes, Storage B the same again)

Remote Copy

The systems are next to each other, one rack to the right, so yes, same DataCenter.
Synchronous Remote Copy over FC.
Every enclousure is connected to both 3PAR Systems. On the MPIO I see only active/active paths.

Hosts

Every Server runs at Windows 2012 R2. We have about 3 Hyper-V Server in a Cluster. All guest systems user virtual disks trough the Hyper-V Host, nothing special configured so far I can see.
Yes, the hypervisors use active/active multipathing with MPIO.

 Best regards

michtsch

[michtsch]

Dear yapkl

We are using 2 AO configurations:

Name: AO_T1-T2
Mode: Performance
Tier 0: SSD_r5
Tier 1: FC_r5

Name: AO_T2-T3
Mode: Performance
Tier 1: FC_r5
Tier 2: NL_r6

Best regards

michtsch

[Sheldon Smith]

It is definitely a mess, but it should be possible to clean up the zoning with production up. It would take some consulting time to do so. You can contact me at michtsch.4.ssmith3278 at spamgourmet.com .

All the new zoning can be laid out on paper, and when it's been double-checked, loaded on the fabrics and committed.
The switches should have their firmware updated:

• v7.2.1d > v7.2.1g > 7.3.1e
• v7.3.1a > 7.3.1e

Then 7.3.1e > 7.4.1e. Did you mention those are 16Gb switches? If so, 7.4.1e > 8.0.2a > 8.1.0a.

If AO jobs are enabled on the (Remote Copy) secondary system, then it's quite possible the secondary volumes have been migrated to the lower tier. With regards to Remote Copy, AO is system-independent; movement on the primary volume does not occur on the secondary volume.