- Community Home
- >
- Storage
- >
- HPE Storage Tech Insiders
- >
- Storage Performance Benchmarking Basics
-
- Forums
-
Blogs
- Alliances
- Around the Storage Block
- Behind the scenes @ Labs
- HPE Careers
- HPE Storage Tech Insiders
- Infrastructure Insights
- Inspiring Progress
- Internet of Things (IoT)
- My Learning Certification
- OEM Solutions
- Servers: The Right Compute
- Shifting to Software-Defined
- Telecom IQ
- Transforming IT
- Infrastructure Solutions German
- L’Avenir de l’IT
- IT e Trasformazione Digitale
- Enterprise Topics
- ИТ для нового стиля бизнеса
- Blogs
-
Quick Links
- Community
- Getting Started
- FAQ
- Ranking Overview
- Rules of Participation
- Contact
- Email us
- Tell us what you think
- Information Libraries
- Integrated Systems
- Networking
- Servers
- Storage
- Other HPE Sites
- Support Center
- Enterprise.nxt
- Marketplace
- Aruba Airheads Community
-
Forums
-
Blogs
-
InformationEnglish
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Receive email notifications
- Email to a Friend
- Printer Friendly Page
- Report Inappropriate Content
Storage Performance Benchmarking Basics
Introduction-
In a universe not long ago in a data center near you, storage benchmarking was a fairly easy affair. Hook up some hosts you have laying around, load up IOmeter, and press go. In most cases you would be able to easily saturate the backend storage and understand how the array performed under pressure. With the industry wide flash revolution, this has become more difficult. Not only are the arrays more performant, but they have a wide variety of data reduction features that can complicate testing. This has led to the popularity of more complicated test tools such as Vdbench, as well as a need for more horsepower in benchmarking rigs. In this blog I will be examining some of the most frequently seen bottlenecks in testing, as well as providing guidance on how to avoid them.
Expectations-
The first step in successful testing is setting your expectations correctly. Make sure you understand the equipment you are testing, and where the limits might be. For example, I would not expect an entry level array to perform at the same level as a high end solution. As such, I would first consult with product experts to determine a ballpark figure for the testing. I have seen this pendulum swing in both directions. Both with expectations being too high and being disappointed by results, as well as being ecstatic with results when the system is only performing at 50% of actual ability. Setting expectations, and testing to them, helps to keep things realistic and to customize systems to meet your needs.
Block sizes come to mind immediately when considering expectations. One misconception I frequently see is that if an array can perform X amount of operations a second, it should be able to meet that number regardless of block size. The truth of the matter is that you should be taking into consideration the testing conditions used to arrive at X. If this measurement was taken using a 4k block size, it would be misguided to assume you could meet the same IOPS count with 128k blocks. When setting expectations around performance, always seek to understand the conditions your reference numbers were based on.
Infrastructure-
The infrastructure you are using to benchmark is just as important as the end device you are testing. Do you use a 10 year old PC to run your applications that require high performance? Then why would you expect it to be able to push a modern storage system to it's limits?
Be aware of the tool you are using to benchmark and the overhead it will require to generate IO. While some tools such as Vdbench are light weight and easily generate IO, other tools that perform tasks such as sending IO transactions through a database stack will require more resources. Consider the tool you are using and the effect it will have on your testing hosts. Generally speaking, you do not want to push the test host past 90% utilization of CPU or memory. If experiencing unexpected results during testing, verify the system resource usage and be sure it is within acceptable boundaries.
Transport infrastructure is also a very important piece of the stack. Having the right amount of connections at the right speed can contribute greatly to the results. If your storage array is capable of multiple GB/s of throughput, how are you going to get the data from point A to point B? Be aware of the theoretical limitations of the technology you are using. Fibre Channel connectivity is known for being fast with low latency, but a single 8Gb port is only capable of driving ~800MB/s of throughput. The same applies for 10Gb Ethernet, a single connection will likely yield around 1,000MB/s.
Understanding the topology of the transport network is also very important. Always evaluate how many switches or pieces of equipment there are in the link, as well as the uplinks between them before you begin testing. Eight connections into the transport will not matter if you only have two uplinks.
Multipathing
With multiple links configured for the transport, understanding how and when each of these links are utilized is key. Whether using Fibre Channel or iSCSI, hosts will have a method of determining the best path to send IO. Windows uses a native multipathing stack with Device Specific Modules or DSMs and Linux typically uses the built in DM-Multipath stack with options set in /etc/multipath.conf. Virtualization hosts will also have settings configured for controlling multipath, such as VMware's Path Selection Policies.
In an optimally configured environment, you should see traffic flowing across all active paths evenly. If only seeing IO use one path, or if paths appear to have uneven amounts of traffic, the first place to look is your multipath settings. Make sure that the multipath configuration is configured to allow traffic across all of the active paths using either a round robin or least queue depth methodology. Consult your vendor's implementation guides for specific settings.
Troubleshooting
You have your beefy test host setup, your transport squared away, your test harness ready.......You press go and immediately are disappointed. You checked with your vendor and they assured you the array would perform to an expectation, but it just isn't getting there. What should you check first?
iSCSI:
Start with the basics. Double check the methodology matches the way your expectation was measured. Verify block sizes, access patterns, read/write ratio, thread count, and data reduction settings.
Check your host utilization, make sure all resources are <90% utilized.
Check your transport and be sure all links are up and negotiating to the right speed, duplex, and MTU. Also a good time to double check topology and uplinks.
Check your number of iSCSI connections per volume. By default this is typically 2, more connections will be needed to drive max performance, I recommend 8 as a starting point. With Nimble arrays this can be configured using Nimble Connection Manager, Nimble Linux Toolkit, or the VMware host plugin.
Verify multipathing and all paths being equally utilized.
If high latency is observed, compare array reported latency vs host reported latency. If host reported latency is off by >.5 ms, investigate host side queues/thread count and transport latency.
Fibre Channel:
Start with the basics. Double check the methodology matches the way your expectation was measured. Verify block sizes, access patterns, read/write ratio, thread count, and data reduction settings.
Check your host utilization, make sure all resources are <90% utilized.
Check your transport and be sure all links are up and negotiating to the right speed. Also a good time to double check topology and uplinks.
Check that your FC queue depths are properly configured for benchmarking. Typically you want these as high as possible. Check your HBA vendor's instructions for how to increase this count in their drivers. Some are set to as low as 32 by default.
Verify multipathing and all paths being equally utilized.
If high latency is observed, compare array reported latency vs host reported latency. If host reported latency is off by >.5 ms, investigate host side queues/thread count and transport latency.
When all else fails, double check the settings again methodically. With all of the different places in the stack to look, it can be very easy to overlook a setting.
- Back to Blog
- Newer Article
- Older Article
- Back to Blog
- Newer Article
- Older Article
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Receive email notifications
- Email to a Friend
- Printer Friendly Page
- Report Inappropriate Content
- HPEStorageGuy on: MSA Best Practices: Upgrading Controller Firmware
- HPEStorageGuy on: What? You’re using a ProLiant DL560 and an MSA Sto...
- HPEStorageGuy on: 3PAR and InfoSight QuickStart guide
- csullivan43 on: Using HPE Predictive Cloud Platform to Address Spl...
- freshcalendar on: Oracle RAC Cloning and Scheduling Snapshots with H...
- commguyaf on: Merged Pool Scale Out with Nimble Connection Mange...
- sleota16 on: Storage Performance Benchmarking Basics
- fgrahn80 on: NOS4 Use Case: Leveraging NWT4 cmdlets for SQL Ser...
- fgrahn80 on: NOS4 Use Case: Rapid Deployment of SQL Developer C...
- jmonger96 on: Nimble OS 4 – Hyper-V VSS Requestor
-
3PAR
14 -
ai predictive analytics
2 -
App Integration
44 -
Cloud Volumes
5 -
Containers
13 -
data protection
3 -
high performance computing
1 -
InfoSight
13 -
MSA
5 -
Nimble Storage
30 -
Oracle
1 -
Oracle Database
6 -
Performance & Data Protection
21 -
SAP HANA
5 -
Setup & Networking
94 -
VMware
13
- « Previous
- Next »
Hewlett Packard Enterprise International
- Communities
- HPE Blogs and Forum
© Copyright 2019 Hewlett Packard Enterprise Development LP