- Community Home
- >
- Storage
- >
- Midrange and Enterprise Storage
- >
- HPE 3PAR StoreServ Storage
- >
- Re: HPE 3PAR 7400 poor performance / designing que...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-02-2017 11:46 PM - edited 04-03-2017 01:13 AM
04-02-2017 11:46 PM - edited 04-03-2017 01:13 AM
HP 3PAR 7400 poor performane / designing question
Hello Guys
I'm a system engineer in my company. We have a strange behaviour with our 3 PAR 7400. So we think it's a strange behaviour :-).
Either, we weren't able to help designing an implementing the system. The system runs about 1 year and the performance is poor (in our opinion, very poor). If we backup (Veeam and HP DP) we have an average troughput from about 150 MB's (as well if we do a normal copy Job on an SSD LUN). I have more speed on my USB Disk. It was an external company who set up everything.
So our job is now reverse engineering :-), what a pleasure.
We have virtual servers and normal server's, everythig is running in two new Blade C7000 enclousures with virtual connect. installed by this company as well. Here's what we got:
2 x C7000 Blade Encloursure
Each Enclousure has:
2 x HP VC FlexFabric 10GB/24-Port Module
2 x HP VC 8Gb FC Module
Then we have:
4 x HP SN6000B 16GB FC Switches
1 x Store Once 4500 with upgrade cage (VTL for Backup)
2 x StoreServ 7400c
6 x DriveShelf M6710
Each Shelf has: 8 x 10K 900 GB SAS, 6 x 72K 1TB NL SAS, 4 x 150K 480GB SSD
(Same 3 PAR Hardware again for RemoteCopy)
Everything is connected with FC. So in my opinion this should run llike hell. But it don't...
As I allready told the Backup is slow, a copy Job from a hardware server to an SSD LUN is slow (150 MB's). If I run a checkwith I/O Meter, sometimes I won't come over 35 MB's with 16 worker (single I/O). I know, 3 PAR is designed for random I/O's, but I'm sure there have to be more performance as well in single I/O's. Reading is sometimes very fast. I did a copy joby (150GB, a Veeam Backup) from the 3 PAR SSD LUN to the Hardware Server multiple times, several times a week. Sometimes I was on 1,5 GB's, most of the time it went down to about 350 MB/s. If I check the 3 PAR during test's on cli, it just sleeps...
I know there are several things the company who implementet the 3PAR did not a good Job:
- Wrong SAN zoning (Server and every target in one zone)
- SAN Switch FW in Fabric different
- RemoteCopy Job from SSD to FC
- Ports on 3PAR not configured symmetrically
Other thing I don't understand they did: We have 2 3PAR System's. Every Blade Center is Connected with each 3PAR. But one 3 PAR is only used for Remote Copy. Is this some kind of best practice? (I mean the cabling)
How have you guys set up your 3 PAR environment? What performance should we be able to expect? What results do you have with I/O Meter? Do you have any advices for us?
We allready read so many best practices guides for the 3PAR and we now, there is a lot we should change. But I wan't to know how you guys do it in your IT and if you have any advices for us? Especially with the cabling of the Remote Copy 3PAR and the Blade Center's I don't understand.
Many many thanks in advance for your help.
Cheerio
michtsch
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-02-2017 11:59 PM - edited 04-02-2017 11:59 PM
04-02-2017 11:59 PM - edited 04-02-2017 11:59 PM
Re: HPE 3PAR 7400 poor performance / designing question
> If I check the 3PAR during tests on cli, it just sleeps
Are you saying when you use statvlun, you don't see much activity?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2017 12:28 AM
04-03-2017 12:28 AM
Re: HP 3PAR 7400 poor performane / designing question
These are complex questions which can't be addressed in a forum. 3PAR best practices document is here: https://www.hpe.com/h20195/v2/GetPDF.aspx/4AA4-4524ENW.pdf
As for your config, based on the drive quantities you gave (24x480G, 48x900G, 36x1TB NL) it's not exactly a small starter config so performance wise should be ok. There are so many considerations:
1. Is this a 7400C 2N or 4N?
2. What is the 3PAR OS version?
3. Are you using any tiering (Adaptive Optimization)?
4. Are the hosts using active-active multipath (MPIO, VMware RR, etc..)
Even on a small minimum config 3PAR with SSD (AFC/AO) I can easily push iometer to 50k IOPS at 100% random, 8k, 70% read workloads and well over 500MB/s for 256k sequential writes.
So like you said, you may have to review every single data path to see where it's slowing you down to 35MB/s.
You mentioned you're using Veeam, if your 3PAR has virtual copy licenses, then you can also do Veeam + 3PAR snapshots integration - that could help a bit. https://helpcenter.veeam.com/docs/backup/vsphere/hp_3par_add.html?ver=95
yapkl
Note: The opinions & comments above are my personal opinions, not of Hewlett Packard Enterprise
----------
If my post was useful, click on my KUDOS! thumb below! Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2017 12:29 AM
04-03-2017 12:29 AM
Re: HPE 3PAR 7400 poor performance / designing question
Jup, you're right. I did the test again with I/O Meter:
1 Worker
32KB 100% Write
1# of outstanding I/Os
cli% statvlun -ni -hostsum -iter 1
09:19:08 04/03/2017 r/w I/O per second KBytes per sec Svt ms IOSz KB
Hostname Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
server1 t 2564 2564 2564 84022 84022 84022 0.33 0.33 32.8 32.8 1
8 Worker
32KB 100% Write
1# of outstanding I/Os
cli% statvlun -ni -hostsum -iter 1
09:20:51 04/03/2017 r/w I/O per second KBytes per sec Svt ms IOSz KBe
Hostname Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
server1 t 12292 12292 12292 402778 402778 402778 0.47 0.47 32.8 32.8 7
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2017 01:00 AM
04-03-2017 01:00 AM
Re: HP 3PAR 7400 poor performane / designing question
hello yapkl
Many thanks for your answer, now at least I have some performance values I can deal with.
So regarding your question:
1. Is this a 7400C 2N or 4N?
It is a 7400C 4N
2. What is the 3PAR OS version?
We are currentrly running on OS: 3.2.1 (MR3)
3. Are you using any tiering (Adaptive Optimization)?
Yes, we use AO
4. Are the hosts using active-active multipath (MPIO, VMware RR, etc..)
Yes, the using active/active multipathing (MPIO)
Cheerio
Michtsch
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2017 07:14 AM
04-03-2017 07:14 AM
Re: HP 3PAR 7400 poor performane / designing question
As yapkl wrote above, "[t]hese are complex questions which can't be [properly] addressed in a forum". That being written, here are some basic observations and questions:
SAN switches.
Firmware. At the time of this writing, 8.1.0a is recommended. The firmware can be updated on-line.
As far as the SAN zoning goes, if the 16GB switches are running fairly current firmware (7.4.x or later), they have added what is called "Peer Zoning". This is in addition to the traditional single-server zoning that has been used for decades.
With a Peerzone, a 3PAR port's PWWN is designated as "principal" and any hosts' PWWNs are designated as "members". A Peerzone isolates the members from each other and only allow communication to the principal. This is part of the mechanism for the HPE 3PAR Smart SAN software. So if the SAN zones are Peerzones, then they may be fine. If they're not, the zones need to be cleaned up. The zoneshow command will display the entire zoning configuration.
Remote Copy.
How far apart are the two systems? Same data center? Miles? Remote Copy over FC or over IP? Synchronous or asynchronous Groups?
Are both blade enclosures adjacent to one system, or is one enclosure close to each system?
Hosts.
What operating system(s) are used? Hypervisors? Virtual guests? Physical servers? Are any guests using a pass-through mechanism to the systems, or do all guests use virtual disks via the hypervisor hosts?
Do the hypervisors have active-active multipath (MPIO, VMware RR, etc..)? I have seen guests with round robin, but the ESXi hosts are still using the default Most Recent Used pathing not Round Robin.
Note: While I am an HPE Employee, all of my comments (whether noted or not), are my own and are not any official representation of the company
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2017 07:04 PM
04-03-2017 07:04 PM
Re: HP 3PAR 7400 poor performane / designing question
If you're doing AO, please review your AO config and make sure you're not doing tiering with NL and using cost mode. That will push a lot of data to NL tiers and when you try to access those, performance will be from your NL tiers (assuming no cache).
https://www.hpe.com/h20195/V2/GetPDF.aspx/4AA4-0867ENW.pdf
yapkl
Note: The opinions & comments above are my personal opinions, not of Hewlett Packard Enterprise
----------
If my post was useful, click on my KUDOS! thumb below! Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-04-2017 04:29 AM - edited 04-04-2017 04:31 AM
04-04-2017 04:29 AM - edited 04-04-2017 04:31 AM
Re: HP 3PAR 7400 poor performane / designing question
Dear Sheldon Smith
Thanks a lot for your answer!
Here my input:
SAN Switches
So I checked the firmware of the SAN Switches. Basically there are 2 Fabrics with 2 Switches each.
Fabric A: Switch1 = v7.3.1a
Fabric A: Swicht2 =v7.2.1d
Fabric B: Switch1=v7.3.1a
Fabric B: Swtich2= v7.2.1d
The zoning is a "mess". I try to explain how they did the zoning:
F.e. Zone for Server1
Server1 (2members)
Storage A Port1
Storage A Port2
Storage A... until Port8
Storage B Port1
Storage B.... until Port8
BackupServer (44 Members)
It becomes more and more clearly that the didn't exactly now what they did. The worst thing of all, this is in production. So we won't be able to just do a new zoning on the fly. Storage A is connected to Fabric A + B and Storage B is connected to Fabric A + B. (Just for the notes, Storage A is for me one 3PAR 7400c with 4 Nodes, Storage B the same again)
Remote Copy
The systems are next to each other, one rack to the right, so yes, same DataCenter.
Synchronous Remote Copy over FC.
Every enclousure is connected to both 3PAR Systems. On the MPIO I see only active/active paths.
Hosts
Every Server runs at Windows 2012 R2. We have about 3 Hyper-V Server in a Cluster. All guest systems user virtual disks trough the Hyper-V Host, nothing special configured so far I can see.
Yes, the hypervisors use active/active multipathing with MPIO.
Best regards
michtsch
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-04-2017 04:41 AM
04-04-2017 04:41 AM
Re: HP 3PAR 7400 poor performane / designing question
Dear yapkl
We are using 2 AO configurations:
Name: AO_T1-T2
Mode: Performance
Tier 0: SSD_r5
Tier 1: FC_r5
Name: AO_T2-T3
Mode: Performance
Tier 1: FC_r5
Tier 2: NL_r6
Best regards
michtsch
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-04-2017 06:49 AM
04-04-2017 06:49 AM
Re: HP 3PAR 7400 poor performane / designing question
It is definitely a mess, but it should be possible to clean up the zoning with production up. It would take some consulting time to do so. You can contact me at michtsch.4.ssmith3278 at spamgourmet.com .
All the new zoning can be laid out on paper, and when it's been double-checked, loaded on the fabrics and committed.
The switches should have their firmware updated:
- v7.2.1d > v7.2.1g > 7.3.1e
- v7.3.1a > 7.3.1e
Then 7.3.1e > 7.4.1e. Did you mention those are 16Gb switches? If so, 7.4.1e > 8.0.2a > 8.1.0a.
If AO jobs are enabled on the (Remote Copy) secondary system, then it's quite possible the secondary volumes have been migrated to the lower tier. With regards to Remote Copy, AO is system-independent; movement on the primary volume does not occur on the secondary volume.
Note: While I am an HPE Employee, all of my comments (whether noted or not), are my own and are not any official representation of the company