Hello,
Please consider uploading configuration information so the group can help. Please also consider opening an HPE support case if your device is covered under support.
To generate text file of the config:
- Using Putty (or other SSH client), open a connection to your MSA and log in.
- Turn off page display pausing by typing 'set cli-parameters pager off'
- Turn on output logging from Putty to a text file
- Run the command 'show configuration'
- Upload the text file to a message on the thread
Note that this file will have information in it about your host names, IP addresses, device name, etc. If you are concerned about sharing that then you can edit out what you are concerened sharing or I can private message you a URL to upload the file to. You can also generate a support log and upload that to the URL I send. Attaching something to this message will allow more people to view and suggest options versus private URL. Note that I will be out of the office EOD today for 9 days. I'll take a look at anything that gets uploaded over the next few hours.
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
Done, please check private mail.
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
File received
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
I've received and reviewed the config information. For the benefit of the group, I'll post a summary so far.
- Check your server host BIOS power profile settings to make sure all hosts are set to "Maximum Performance". This is the cause of 90% of the storage "performance issues" I have seen in the past.
- Looking at the MSA config itself, it looks pretty good. Everything looks healthy and you've done a good job keeping your RAID set sizes identical and to a power of 2.
- One thing you have done, which is not best practice, is that you have 10 and 15k disks in the same pools and they are different sizes.
- The disks in your 15k vdisks are totally full (as would be expected), the 10K pools have an even amount of space which is good and means they are balancing out
- This will cause all new writes to go only to the larger 10k disks. Reads will be unpredictable. Older data will come from both the 15k and 10k disks, newer data may only come from the 10k disks. While this isn't best practice, I'm not sure it's the cause of your issue. I would lean towards host power profile settings, zoning, or multipathing issues. Please check those items.
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
Any update on this situation following the suggestions I provided?
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
A healthy 8GB fabric will do just fine. 16GB is faster (obviously) and overall more efficient but I've seen very heavy workloads do just fine on 8GB (and 4GB for that matter).
Here is how the MSA defines a "tier"
- Performance=SSD
- Standard=15k or 10k
- Archive=7.2k
You can't change this. Therefore; 10 and 15k drives, in the same pool, will been seen as the same tier. IO's will therefore have the potential to perform at the lowest common denominator. In this case 10k. Also, if you have different RAID types or set sizes in the same tier (within a pool) then IOs would porentially operate at that lowest common denominator as well.
We'd really need to dig into your "workload" more. How are sure that the workload is identical between your environments? What's your percentage of read/write? Block size? etc?
Can you supply me with some performance info? Like outputs from:
- show controller-statistics
- show host-port-statistics
- show disk-group-statistics
- show disk-statistics
Also, if you haven't already, you may want to open a support case.
Thanks.
I work for HPE. The comments in this post are my own and do not represent an official reply from the company. No warranty or guarantees of any kind are expressed in my reply.
@oli_4 I have looked at the data that you shared but they are just single output from which we can't conclude anything. However Controller A CPU usage looks high but rest all data looks fine.
1st of all we shouldn't compare MSA2040 with P2000..........one is Linear array and another one is Virtual Array.
In order to troubleshoot Performance issue, there are many factors involved and it's not straight forward task. Some of the best practice to follow can be no hardware issue should exist, firmware need to be up to date, Connected system like Servers, SAN Switch all need to be up to date with driver/firmware as well.
Need to check what is the block size set at the Host and depends on that we should check if you want high IOPs or high throughput. Specifically, the smaller the I/O size, the more I/Os per second (IOPS) the SAN can process. However, the corollary to this is a decrease in throughput (as measured in MB/s). Conversely, as I/O size increases, IOPS decreases but throughput increases. When an I/O gets above a certain size, latency also increases as the time required to transport each I/O increases such that the disk itself is no longer the major influence on latency.
Typically, workloads can be defined by four categoriesтАФI/O size, reads vs. writes, sequential vs. random, and queue depth.
A typical application usually consists of a mix of reads and writes, and sequential and random.
For example, a Microsoft┬о SQL Server instance running an OLTP type workload might see disk IO that is 8k size, 80 percent read, and 100 percent random.
A disk backup target on the other hand might see disk IO that is 64k or 256K in size, with 90 percent writes and 100 percent sequential.
The type of workload will affect the results of the performance measurement.
Check this below Customer Advisory and disable "In-band SES" ,
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c05306564
You can check the below Customer Advisory as well.........in many situations this helped to improve performance,
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03473698
If you have specific requirement and you want only SSD pages to deal with your IO then use 'Tier Affinity' on the particular volume.
Check for ODX settings in Windows system. As per SPOCK it clearly says that Microsoft Offline Data Transfers (ODX) is not supported with MSA2040. You check another Forum -> https://community.hpe.com/t5/MSA-Storage/Is-MSA-2040-Certified-for-Windows-ODX/td-p/6967522
https://h20272.www2.hpe.com/SPOCK/Content/ExportPDFView.aspx?Id=91936&typeId=2
Tried to give you step by step SPOCK links along with applied filter.
Download the PDF and in page 4 you will get it's clearly mentioned that Microsoft Offline Data Transfers (ODX) is not supported
If you still face performance issue then at the time of performance issue happening capture the below outputs atleast 10 to 15 times along with MSA log and log a HPE support case. They will help you.
# show controller-statistics
# show disk-statistics
# show host-port-statistics
# show vdisk-statistics
# show volume-statistics
@IT_Prof Not much data available to check from MSA perspective. So difficult to tell what is wrong in your situation.
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Difficult to provide any recommendation without verifying entire setup.
As per the screenshot I see both Controller CPU usage very low so this looks fine. However I see all drive IOPs more than 200 but I don't know the drive model number so difficult for me to check if this is crossing threshold limit or not.
Another thing noticed all IOs seems to be happening through Host port A1 so you need to verify multi-path policy at the Host system. It seems you have fixed path policy set which need to modify as round-robin
You can always log a HPE Support case as well to get basic performance analysis.
Hope this helps!
Regards
Subhajit
I am an HPE employee
If you feel this was helpful please click the KUDOS! thumb below!
*********************************************************************
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Thank you for sharing the model number of the drive and according to that this is Midline SAS drive which means Safe IOPs value approx 75 only but all of the drives shows more than 200 IOPs
It means high load on the array causing Performance issue. You need to control the load may be reduce backup jobs from VEEAM to fix this.
Hope this helps!
Regards
Subhajit
I am an HPE employee
If you feel this was helpful please click the KUDOS! thumb below!
*********************************************************************
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]