StoreVirtual Storage
1748089 Members
4873 Online
108758 Solutions
New Discussion

Re: StoreVirtual VSA : Misconfiguration ? Bad write performances

 
Nask
Occasional Advisor

StoreVirtual VSA : Misconfiguration ? Bad write performances

Hello,

I'm trying to setup a simple HPE StoreVirtual VSA infrastructrure :
This is a lab, not a production, just for training stuff.

HV01:
Bi-Xeon L5630 - 32GB Mem.
HDD0: 1x 1TB - HDD1: 1x 2TB
NIC: 2x1 GbE + 2x 10GbE (1 used)

HV02:
Bi-Xeon L5630 - 32GB Mem.
HDD0: 1x 1TB - HDD1: 1x 2TB
NIC: 2x1 GbE + 2x 10GbE (1 used)

HV01 and HV02 got their 1GbE connected to a switch.
HV01 and HV02 got their 10GbE connected each other with a DAC. (1 used)

10GbE : 10.10.10.0/28
1GbE: 192.168.1.0/24

HV01:
10GbE: 10.10.10.3
HPE VSA DSM has been installed.
  VM SRV-VSA01 (on HV01):
  IP: 10.10.10.1
  Jumbo framed configured at 9000.

HV02:
HPE VSA DSM has been installed.
10GbE: 10.10.10.4
  VM SRV-VSA02 (on HV02):
  IP: 10.10.10.2
  Jumbo framed configured at 9000.

Both Hyper-V servers can communicate between each other and with VSA virtual machines through the 10GbE adapter.

CMC has been deployed on HV01, I created a Management Group which contains SRV-VSA01 and SRV-VSA02.
I created a VSA cluster which contains SRV-VSA01 and SRV-VSA02.
I created a Network-Raid 10 volume.
I added SRV-HV01 and SRV-HV02 as servers, created a server cluster.
I assigned the previous created volume to this server cluster.

Once done, I just had to refresh ISCSI targets and get the freshly created volume.
I just had to put it online, initialize it,  create a new volume, format it as ReFS, assign a letter.

And here we are now. I'm faced with some performances issues and I have to say, I'm not that quite confident about my setup / configuration.

When I run a CrystalDiskMark I have extremely low write speed.

Results when I run CrystalDiskMark (5/1GB) on the hard-drive directly on the HyperV:

SRV-HV01
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 179.562 MB/s
Sequential Write (Q= 32,T= 1) : 157.042 MB/s
Random Read 4KiB (Q= 32,T= 1) : 1.580 MB/s [ 385.7 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 1.553 MB/s [ 379.2 IOPS]
Sequential Read (T= 1) : 67.319 MB/s
Sequential Write (T= 1) : 65.847 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.235 MB/s [ 57.4 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.552 MB/s [ 134.8 IOPS]

Test : 500 MiB [G: 0.7% (12.9/1862.9 GiB)] (x5) [Interval=5 sec]
Date : 2016/12/05 1:27:24
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

Results when I run CrystalDiskMark (5/1GB) on the volume presented by VSA (Network-Raid10):

SRV-HV01
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 289.112 MB/s
Sequential Write (Q= 32,T= 1) : 23.408 MB/s
Random Read 4KiB (Q= 32,T= 1) : 4.776 MB/s [ 1166.0 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 0.440 MB/s [ 107.4 IOPS]
Sequential Read (T= 1) : 182.402 MB/s
Sequential Write (T= 1) : 11.324 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.946 MB/s [ 231.0 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.098 MB/s [ 23.9 IOPS]

Test : 500 MiB [E: 0.0% (0.1/299.9 GiB)] (x5) [Interval=5 sec]
Date : 2016/12/05 1:34:16
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

Results when I run CrystalDiskMark (5/1GB) on the volume presented by VSA (Network-Raid0):

SRV-HV01
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 353.499 MB/s
Sequential Write (Q= 32,T= 1) : 36.200 MB/s
Random Read 4KiB (Q= 32,T= 1) : 5.495 MB/s [ 1341.6 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 1.080 MB/s [ 263.7 IOPS]
Sequential Read (T= 1) : 199.443 MB/s
Sequential Write (T= 1) : 13.633 MB/s
Random Read 4KiB (Q= 1,T= 1) : 1.001 MB/s [ 244.4 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.116 MB/s [ 28.3 IOPS]

Test : 500 MiB [F: 0.2% (0.1/49.9 GiB)] (x5) [Interval=5 sec]
Date : 2016/12/05 1:40:31
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

Results on SRV-HV02 are quite similar.

 

4 REPLIES 4
Michal Doležal
Frequent Advisor

Re: StoreVirtual VSA : Misconfiguration ? Bad write performances

If I understand well you have just one drive for VSA (2TB drive? yes?). Read is very good, write is bad. In StoreVirtual you are saving/striping blocks to all nodes, in case you have NRAID10 you are saving two copies of blocks of DATA.  SATA drives (i expect that is traditional SATA) can have overal performance close to 100 IOPS. And this can be answer especially when we look for 4k random.

1. You are testing two nodes connected together with 10G network, so network looks probably not the issue. BUT normally you are configuring jumbo frames for HW version of the StoreVirtual (4330/4530 etc), I am not exactly sure if usage of jumbos are fully supported for VSA.

       a. So problem can occur due to inconsitent jumbos. (VSA, HV-SWITCH, NIC card) try without it

       b. Problem can occur due to direct connection between computers (for StoreVirtual 4k not supported, for VSA in this setup can be used)

      c. Take your attention to VMQ settings and especially eventlog - VMQ in Hyper-V is long story.

Check NICs how much datas are transfering, check perfmon for internal vSwitch datas.

2. But by My opinion is that the speed bad due to just one drive and due to character of write operation in StoreVirtual. Did you try to take a look to CMC? Take a look on performance tab and see what shows. All info is important, but one the most, Queue lenght.

3. try setup without DSM, I think this can be in this cfg also small issue. Try connection to just one node. (this can show where problem is, if this is OK, then problem is in replication link between both SVs.)

Can you post the results from CMC, from hyper-V perfmon (I/O statistics) and from vSwitch and Nics?

 

 

Michal Dolezal, DiS.
System engineer
AVE BOHEMIA, s.r.o.
Nask
Occasional Advisor

Re: StoreVirtual VSA : Misconfiguration ? Bad write performances

Hello,

Thanks for your answer, I'll try all things you said and get back to you as soon as possible.


Even though I posted some CrystalDiskMark results, which they are not really meaningful, I do have tried different IOMeter setups.
100% Random - 100% Write - 8 KB ( + Req. size boundaries)
Results directly on the 2TB drive: ~ 170 iops, 1.38MB/s, avg time: 23ms.
I had quite different results, but in the end, for the presented volume: ~23 iops, 0.19 MB/s, ~168ms.

If you have any much more interesting profile I could try, tell me.

Nask
Occasional Advisor

Re: StoreVirtual VSA : Misconfiguration ? Bad write performances

Hello again,  I've been able to do some tests, just as you said :

- Disabled Jumbo Frames : No changes. I tried disabling Jumbo Frames on 10GbE and I also tried to switch on 1x GbE, it didn't change anything.
I also tried using a switch in-between, it didn't change.

- Disabling VMQ : No changes. Did on 1 GbE and 10 GbE.

- Uninstalled VSA DSM MPIO : No changes.

Whatever I could do on this setup, it doesn't change, as long as I use the volume presentend by VSA, writes are extremely slow.

Nask
Occasional Advisor

Re: StoreVirtual VSA : Misconfiguration ? Bad write performances

I've been able to test one more thing :

- Disabled Write-Cache : No changes. I disabled the Write Cache on the 2 TB drive, on the volume presented by VSA, but I couldn't disable where there is the VM VSA, the system hard-drive.