MSA 2012i DC : slow read (but good write speed)

[inti]

Hi HP-Folks and Users,

we got quite a strange issue with a dual controller MSA2012i. Basically the read speed is crawling slow (about 1-3 MB/sec) while writing data yields good results (about 40-100+ MB/sec depending on filesize). This behavior is the same on all vdisks/volumes and from any of the three connected servers.

Vdisk Configuration is: one Vdisk per Controller (4x450GB@Raid5 and 3x450GB@Raid5 + global Hotspare) and two volumes on the big and one volume on the smaller vdisk, each volume is in access by one server. All Drives are SAS-Drives. All connected Servers are Windows 2003 Machines accessing with MS iSCSI-Initiator (v2.08).

The Configuration is like the recommended Setup from the Manual, that is both Port0s in a subnet and both Port1s in a Subnet, both connected via separate (Layer2) switches to the servers which also have 1 Port in each Subnet. These switches only serve iSCSI-Traffic. All four Controller Ports are connected in MS iSCSI-Initiator (2.08) and MPIO is set up with RoundRobin Policy for each connected volume (confirmed working with troughput-measuring). We also installed the MSA2000 VSS/VDS/Capi Packages along with the three recommenced Hotfixes on each server.

Each NICs with iSCSI-Access on all servers is configured with:

Jumbo-Frames @ 9k
TCP Offload Engine enabled
TCP AckFrequency set to 1 like mentioned in the Guide
LinkDownTime set like in the Guide
Set to fixed 1Gbit Link Speed

Host Ports on the MSA2012i are also set for Jumbo-Frames and 1Gbit Link Speed.
Patched latest Firmware JP210P12 .. (which was quite troublesome from JP210P03 but worked in the end). Write-Back cache is enabled and Disk-Drive Read Cache enabled on all drives.

We got not a single CRC/Transmission Error on any of the iSCSI-Network connections.
Tried Read-Ahead-Cache with Default, 4 MB and Maximum Setting .. no difference.

Basically it is just the low read speed from the MSA2012i which causes trouble.
Has anybody seen a similar issue .. or better: any ideas ?

[Pedro Eleuterio · Accepted solution]

I had EXACTLY the same problem this week (I have exactly the same configuration you have!). I solved it yesterday, I bet it's the Jumbo Frames at 9k. I also had very slow reads, but ok writes, BUT normal reads if I was copying between volumes on the same server. If you try to read data from MSA, from a network connected computer, it will have to use other switches. If those do not support 9k Jumbo frames, but only 4k, they will have to break them...and I thinks that's the reason why it's so slow. Just use the default 4k Jumbo frames on you iSCSI Nics, and everything will be ok.

I also upgraded both controllers firmware to the latest 16 March JP210P12...but I had to disable firmware communication between the controllers, on system configuration. Before I did that, the controller with the older firmware would overwrite it on the newest controller firmware! I patched it on both controllers, one at a time. This firmware fixes a bunch of problems, I would highly recommend it.

[inti]

Thank you.

I just tried it this morning and it DID solve the weak read rate. Stange, as all components (MSA, Switches and NICs) are said to support 9k-Jumboframes. We also tried other switches now (1800-8G .. which are also used in the MSA200i-Starter-Bundle) .. was no differece.

As soon as Backup is complete, I'll just have to switch MPIO back on (had it off for debugging) and everything should be ok.

Well I guess I'll just keep Standard-MTU in the future.

I will report back after completion.

P.S.
tbh .. these 1800-8G are a bit of a joke .. no rackmount possible and external powersupply .. and the next size is 1800-24G priced at about ut 5x the price .. no comment.

[inti]

Ok it works now. Hovever just switching back to 4k-Jumbo didn't do the trick (better speed, but still staggering). I needes to switch it off completely. Maybe this has someting to do with flow-control and/or the "simple" switches.

Now I don't get the same write rates for sequentiel writes (before: more than 100 MB/s, not about 55MB/s .. both times with MPIO) but that is something for research in the future.

I nog get about ~60MB/s with one thread over one or two pathes. If I use multiple Threads it goes to 100MB/s+ . so thats "normal" I suppose.

[Pedro Eleuterio]

We have HP 1400-24G switches. They have no management but are cheap, and recommended for iscsi.

I found out yesterday a very weird problem this week. I cannot do Microsoft Shadow copies on a MSA2000i volume on my Windows Server 2003 R1 SP2...?! I dont know why, I have tried everything. I just receive this error in system log:

Event Type: Error
Event Source: VolSnap
Event Category: None
Event ID: 28
The shadow copy of volume L: could not be created due to a failure in creating the necessary on disk structures.

and a message with something about "not enough space on the volume". The 1.5 Tb volume is almost empty! this does not happen on other (not iscsi) volumes. I uninstalled MSA 2000 VSA provider, did not work. I dont know what else to do.

[zuro]

Hello averybody i had same problem with MSA 2012i good writes about 80 mb/s but wery slow reads generally in 1-5 mb/s.

My config is only one controller, both ports are in same subnet, iscsi traffic had own Vlan, switch is blade cisco 3520. MSA is connected to two hyper-v boxes both with two physical nic. In the eventlog I achieve error log Iscsiprt eventid 20 and 7 check attachment.

Should be controller ports in different subnet or can be in the same, if I have only single controller?

[pviellieux]

Is anybody getting a resolution on this? I cannot get more than 20 MB/s, and more typical is 3-7 MB/s. I tried the Jumbo frames and that was just a mess. All kinds of errors and even brought down my servers! (iSCSI on separate LAN/vLAN/Switches, too)

I see lots of other posts complaining about "only 110 MB/s" or even 40-50... I would be thrilled with that.

[inti]

Well basically turning off Jumbo-Frames on Jumbo-Frames with the "cheap" Switches or replacing the Switches with more "intelligent Devices (probably because of bigger buffers) while leaving Jumbo-Frames on solved the issue for me.

Do you get these Values per Thread or at all and only Read/Write or both.