- Community Home
- >
- Storage
- >
- Midrange and Enterprise Storage
- >
- HPE EVA Storage
- >
- Re: EVA 4400 disk performance
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2012 12:48 AM
10-23-2012 12:48 AM
EVA 4400 disk performance
Hi
We're running HP blade system on Hyper-V 2008 R2 cluster with EVA 4400 storage.
EVA has two disk groups
- diskgroup 0 (enclosuer 0) with 10 FC disks 400GB 10k
- diskgroup 1 (enclosure 1) with 8 FC disks 300GB 15k
We're dealing with a high impact on disk I/O performance on SQL servers running on Hyper-V hosts on VM (Win2008 + SQL 2008).
R/W performance is between 20-30MB/s, but only on servers running SQL. On all other VM servers without SQL performance is between 50-60MB/s.
We have already consulted with Microsoft support, they didn't find any issues with cluster or VM, also there is no other HW impact on performance.
The only conclusion by our appinion is that SQL servers are generating so many I/O requests, that EVA or HBA on hosts can't handle that.
Is this true, or how can we check that ? SQL profiler show normal activity, also there is no other issues with operating sistem, so we're certain that there must be some kidn of connection between VM - HOST - EVA.
Also running RW test directly on a host connected to EVA, Read results are between 200-300MB/s, but Write is between 60-100MB/s.
Any additional points how to proceed next are welcome.
Regards,
Mike
- Tags:
- Hyper-V
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2012 10:30 AM
11-04-2012 10:30 AM
Re: EVA 4400 disk performance
Hello,
what raid level do your luns use? Generally you have really small number of disks there. But - if your luns perform directly ~300MB/s in read and 100MB/s in write and under virtualisation layer it it so different the problem is virtualization layer very probably...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-05-2012 05:29 AM
11-05-2012 05:29 AM
Re: EVA 4400 disk performance
LUN us RAID5, we did some testing also on RAID1 LUN but the performance is also low.
Yes, at first we thought that the problem is virtualization layer, but then we did some testing on DAS an iSCSI disks and we got higher RW as on SAN.
We also did testing on different virtualization platforms (Hyper-V 3.0, VMware 5) with the same results.
I think that there is a high IO from virtual machines to SAN, so that the EVA can't handle it, because moving the same VM on a DAS it gets higher RW ratio.
Is there any way to check how many IO requests come from VM throught Hypervisor to SAN ? And how many of them can EVA handle ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-08-2012 03:41 AM
11-08-2012 03:41 AM
Re: EVA 4400 disk performance
On the VMware level you can use ESXTOP to analyze the IO your VMs are requesting:
VMware KB: Using esxtop to identify storage performance issues
On the EVA level you should use "Performance Data Collector" (formerly HP Command View EVAPerf )
EVAPERF should already be installed alongside your Command View installation.
Just start evaperf and take a look at the parameters - it's pretty self-explanatory.
evaperf vd -cont 5
for example shows the current statistics of all VDisks (vd) and refreshes every 5 seconds (-cont 5).
Please remember though that this is the sum of ALL IO on this vdisk, to narrow down VM-specific IO loads you need to ensure that only 1 VM is running on this VDisk/Datastore.
Other parameters that might be of interest for you are: vdrl vdwl (latencies)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2012 01:29 AM - edited 11-12-2012 01:55 AM
11-12-2012 01:29 AM - edited 11-12-2012 01:55 AM
Re: EVA 4400 disk performance
Hi WissfeldA
Thank you for valuable informations.
We have 2 disk groups/datastores. Datastore0 has 11 VM, and Datastore1 has 4 VM. Eachn VM is running on its own LUN.
So if I got whis right, the best practice will be to shut down all VM and then start-up one by one and collect informations about storage performance with
- evaperf vd, vdrl, vdwl ?
I have already run evaperf with all the parameters, and here are results below. In time of collecting data, all VMs are running. I copied only VM that are causing 'high' R/W? Are these numbers below normal or critical ?
Regards
evaperf vd
ID Read Read Read Read Read Read Write Write Write Flush Mirror Prefetch Group DRM Online Mirr Wr Ctlr Redundancy NumberOf Preferred LUN Node
Hit Hit Hit Miss Miss Miss Req/s MB/s Latency MB/s MB/s MB/s ID Group To Mode Presentations Path
Req/s MB/s Latency Req/s MB/s Latency (ms)
(ms) (ms)
-- ----- ------ ------- ----- ------ ------- ----- ------ ------- ------ ------ -------- -------------------- --------- ------ ---- ---- ---- ---------- ------------- ------------- --------------------------------------- ----------
13 39 2.50 0.2 174 7.92 0.9 0 0.00 0.0 0.00 0.00 2.07 Default Disk Group 1 Nonmember A Yes Back R026 vraid5 3 no_preference Cluster_005_DG1 EVA001
0 114 7.44 0.5 171 4.85 5.1 2 0.01 0.1 0.06 0.06 8.64 Default Disk Group Nonmember A Yes Back R026 vraid5 3 no_preference Cluster_003 EVA001
evaperf vdrl
ID 0 1.6 3.3 6.6 13 26 52 105 210 420 Ctlr LUN Node
< < < < < < < < < <=
1.6 3.3 6.6 13 26 52 105 210 420 INF
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
14 40 2 8 3 0 0 0 0 0 0 R026 Cluster_009_DG1 EVA001
13 820 16 24 10 4 0 0 0 0 0 R026 Cluster_005_DG1 EVA001
0 1246 78 224 334 31 9 0 0 0 0 R026 Cluster_003 EVA001
evaperf vdwl
ID 0 1.6 3.3 6.6 13 26 52 105 210 420 Ctlr LUN Node
< < < < < < < < < <=
1.6 3.3 6.6 13 26 52 105 210 420 INF
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
14 200 185 508 30 0 0 0 0 0 0 C09V Cluster_009_DG1 EVA001
14 236 352 314 0 0 0 0 0 0 0 R026 Cluster_009_DG1 EVA001
0 87 1 0 0 0 0 0 0 0 0 C09V Cluster_003 EVA001
0 41 0 0 0 0 1 0 0 0 0 R026 Cluster_003 EVA001
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2012 04:13 AM
11-12-2012 04:13 AM
Re: EVA 4400 disk performance
Hi Mike,
just to clarify things ...
- You're running 2 EVA diskgroups called "Default Disk Group" and "Default Disk Group 1"
Which one is the 8x15k, which one the other ? - Those two EVA diskgroups hold different vDisks - one holds 11, the other one 4 ...
Which Raid-Levels do you use there ? "Cluster_003" and "Cluster_005_DG1" are in vRaid 5, the other ones too ? - Do you currently use vSphere or Hyper-V ?
According to your logs, i do not see any unusual entries.
Can you please post the following as well : EVAPERF PDA ; EVAPERF PDG; EVAPERF VDTS ?
Maybe we're able to pinpoint stuff further from here ... although i also suspect the low # of spindles being the root cause for low write throughput and/or write IO/sec.
And please also have a look at the following perfmon counters on your problematic servers as well :
- Physical Disk: Avg. Disk Queue Length (Read Write as well)
- Physical Disk: Disk Reads/sec - Writes/sec - Transfers/sec
- Pyhsical Disk: Split IO/sec
Kind regards
Alex
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2012 04:30 AM
11-13-2012 04:30 AM
Re: EVA 4400 disk performance
Hi WissfeldA.
Thank you for you reply and informations. Here are some additional informations
1. Default Disk Group 10 FC disks 400GB 10k
Default Disk Group1 8 FC disks 300GB 15k
2. Default Disk Group holds 10 vDisks
Default Disk Group1 holds 4 vDisks
All vDisks are RAID 5
We did some testing with additional vDisk on RAID 1, but performance is not been increased
3. We use Hyper-V 2008 R2 (3 node cluster).
We did some testing on vShpere 5, performance was quite the same as on Hyper-V
Reports from evaperf
PDA
Enc. Bay__1 Bay__2 Bay__3 Bay__4 Bay__5 Bay__6 Bay__7 Bay__8 Bay__9 Bay_10 Bay_11 Bay_12 Bay_13 Bay_14 Node
MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s
---- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ----------
2 0.26 0.32 0.44 0.38 0.33 0.36 0.24 0.25 - - - - - - EVA001
1 1.99 2.26 2.27 2.46 2.60 2.66 2.25 3.06 2.33 2.78 - - - - EVA001
PDG
Disk Average Average Average Average Average Average Average Average Number Ctlr Node
Group Drive Drive Read Read Read Write Write Write of
Queue Latency Req/s MB/s Latency Req/s MB/s Latency Disks
Depth (ms) (ms) (ms)
-------------------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ---- ----------
Default Disk Group 0 - 0 0.00 0.0 0 0.00 0.0 10 C09V EVA001
Default Disk Group 3 - 46 0.94 4.1 5 0.25 25.9 10 R026 EVA001
Default Disk Group 1 0 - 0 0.00 0.0 0 0.00 0.0 8 C09V EVA001
Default Disk Group 1 7 - 63 3.16 10.2 50 3.66 19.7 8 R026 EVA001
VDTS
ID 0 2K 4K 8K 16K 32K 64K 128K 256K 512K Ctlr LUN Node
< < < < < < < < < <=
2K 4K 8K 16K 32K 64K 128K 256K 512K INF
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
3 0 0 1 0 0 0 0 0 0 0 C09V 6001-4380-024D-3B98-0000-7000-0005-0000 EVA001
3 0 0 2 0 0 0 0 0 0 0 R026 6001-4380-024D-3B98-0000-7000-0005-0000 EVA001
11 0 0 0 0 0 0 0 0 0 0 C09V 6001-4380-024D-3B98-0000-7000-0015-0000 EVA001
11 0 0 0 0 0 0 0 0 0 0 R026 6001-4380-024D-3B98-0000-7000-0015-0000 EVA001
12 2 0 10 3 1 0 1 0 0 0 C09V Cluster_001 EVA001
12 0 0 6 12 1 0 1 0 0 0 R026 Cluster_001 EVA001
10 0 0 5 0 0 0 0 0 0 0 C09V Cluster_002 EVA001
10 0 0 3 1 1 0 0 0 0 0 R026 Cluster_002 EVA001
5 0 0 0 0 0 0 0 0 0 0 C09V Cluster_004 EVA001
5 0 0 0 0 0 0 0 0 0 0 R026 Cluster_004 EVA001
7 0 0 2 0 0 0 0 0 0 0 C09V Cluster_006 EVA001
7 0 0 2 1 0 0 0 0 0 0 R026 Cluster_006 EVA001
2 0 0 0 0 4 1 0 0 0 0 C09V Cluster_007 EVA001
2 0 0 0 0 4 0 0 0 0 0 R026 Cluster_007 EVA001
15 0 0 0 0 0 0 0 0 0 0 C09V Cluster_008 EVA001
15 0 0 1 0 0 0 0 0 0 0 R026 Cluster_008 EVA001
6 0 0 0 0 0 0 0 0 0 0 C09V Cluster_010 EVA001
6 0 0 0 0 0 0 0 0 0 0 R026 Cluster_010 EVA001
14 0 1 2 20 2 29 545 0 0 0 C09V Cluster_009_DG1 EVA001
14 0 0 2 66 6 75 1754 3 0 0 R026 Cluster_009_DG1 EVA001
4 18 5 1 1 1 0 0 0 0 0 C09V Cluster_011 EVA001
4 17 3 3 8 0 0 0 0 0 0 R026 Cluster_011 EVA001
13 0 0 1 1 0 0 0 0 0 0 C09V Cluster_005_DG1 EVA001
13 0 0 1 0 0 0 0 0 0 0 R026 Cluster_005_DG1 EVA001
0 0 0 5 1 0 2 0 0 0 0 C09V Cluster_003 EVA001
0 8 34 647 148 135 27 124 0 0 0 R026 Cluster_003 EVA001
9 0 0 0 0 0 0 0 0 0 0 C09V Cluster_Witness EVA001
9 0 0 0 0 0 0 0 0 0 0 R026 Cluster_Witness EVA001
1 1 0 1 1 0 0 1 0 0 0 C09V Cluster_012_DG1 EVA001
1 0 0 3 0 0 0 1 0 0 0 R026 Cluster_012_DG1 EVA001
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2012 12:57 AM
11-14-2012 12:57 AM
Re: EVA 4400 disk performance
Hi Mike,
as far as i see there are no obvious problems - but those EVAPERF captures only show a brief timeslice.
As a matter of fact your EVA is running in almost the lowest possible configuration with 8-10 pDisks per DG which tends to be limited in terms of IO/.
Although i do do not see the EVA being saturated with request in your samples, i highly assume the # of pDisks being the main problem.
You're running 10 spindles (in vRaid5) which leads to "theoretical SAFE IOPS of 600/sec". You're hosting 10 LUNS on this diskgroup which increases the overhead a little bit .. my guess is that your "Default Disk Group" is capable of ~500 IOPS in total for all VMs as a realistic estimate. (Default DiskGroup1 should even be a little bit lower, for an explanation google: HP_EVA_Calculator_v3.xlsx)
On the other hand i'm not quite sure what the actual problem is ... do you suffer from performance problems ?
If yes, then how do they show up ?
I can easily flood a large EVA array with 512b I/O leading to a "R/W bandwidth" of only a couple of MB/Sec.
So please understand that " [...] R/W performance is between 20-30MB/s, but only on servers running SQL [...]" may not be enough. How did you measure this R/W performance (simple cut&paste or using IOMeter/<insert tool here>) ?
Also please have a look inside of the problematic VMs' PERFMON counters, especially the "physical drive Queue" and probably the IO counters mentioned in my last post.
Since i'm not very skilled on Hyper-V, i do not know how to gather those on the Hyper-V cluster, especially when those are in Core Installation mode, but those WMI counters should be present on the hypervisors as well.
Sorry that i could not help you quicker, but it's sometimes quite tedious without having "hands-on" ...
Regards
Alex
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-15-2012 12:23 AM
11-15-2012 12:23 AM
Re: EVA 4400 disk performance
Hi Alex
Thanks again for all informations and points. You have give me some clues about diskgroups and disk IO throughput. Thanks !
We didn't use IOMeter, we use SQLIO tool with parameters below.
Can you reccommend us any other tool that we could use for testing ?
sqlio.exe -kR -t50 -s30 -f64 -b32 -i64 testfile.dat
Running 50 threads for 30 sec from 1GB testfile.dat using 32KB IOs over 2048KB stripes with 64 IOs per LUN
Doing the same test with 2KB IOs the results are 10x lower.
Since MS SQL is doing best on a 8KB, we did some testing also, but the result were below 1MB/s for write, and about 2-3MB for read.
We also presented a blank LUN to a host and did the same test on it, but the results were the same.
It looks like that all VM running on cluster are exceeding 500 IOPS that EVA can handle, but on the other hand EVAperf is not showing any difficulties with that.
During the weekend we wil try to shut down all VM on cluster and do some testing on EVA, and then try to run VM one-by-one and do testing if any of this VM is causing such an impact on disk performance.
I know it's a show in the dark, but we'll give it a try, what do you think ?
Thank again for all your help and knowledge.
Regards,
Miha