iscsi initiator poor performance

[Dejan Donovski]

We have tested HP-UX software iSCSI initiator with EMC cx3-20c target. Although HP-UX iSCSI initiator is not supported it works except for the performance problem. We were not able to achieve more then 20MB/s on a iscsi vxfs file system. This is very poor. I could not find any docs regarding iscsi initiator optimization and the only configurable iscsi parameters are authentication related.
The iscsi stats are attached, however i am not able to interpret them. The only suspicious parameter is I/O underflows.

Any advices

[Britto Sidhan]

Hi

I don't know how much the following article will be useful for you. Just check it:

hxxp://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/9687/30571/01410738.pdf

For this you need an userid and password in ieeexplore.

Thanks and Regards
Britto Sidhan

[rick jones]

I would start checking TCP and link-level stats. Netstat -s -p tcp snapshots before and after your transfer, then run through ftp://ftp.cup.hp.com/dist/networking/tools/beforeafter.c

Look for things like retransmissions, and retransmission timeouts and out of order segments.

Do the same thing with lanadmin -g mibstats before and after and look for errors, discards and _late_ collisions.

Also, what sort of system is this, and what is the per-CPU utilization of each CPU while the transfer is taking place? I'm guessing this is a gigabit/s ethernet, but any idea how many I/Os can be outstanding at one time, and of what size?

What is the TCP window size being used - you want to get a packet trace starting from when the target is first mounted so you can see the window scaling option if it is present.

[Martha Mueller]

Now it is a year later, and our company is considering installing an EMC clariion that would be used by two servers: an rx3600 and an rx4640, both running HP-UX 11.23 (that's 11i V2) over iSCSI.

Can anyone comment on this configuration? Found this HP web site, "The following third-party storage devices have completed independent interoperability validation under the guidance of HP and have been qualified as interoperable in HP-UX environments."

http://www.hp.com/products1/serverconnectivity/mass_storage_devices.html#EMC

But it appears that only HP-UX 11iV3 is listed.

I haven't found anything that discusses integrity servers running HP-UX iSCSI with Auto Port Aggregation. Would aggregating two ethernet ports work well with iSCSI?

Has anyone tried this arrangement, am I doomed to have the same performance issues that Dejan had last year? Will there still be performance issues (or any other issues) if we attempt to use iSCSI even without aggregating ports for data? We'll keep network traffic separate from data traffic.

Thanks

[Dejan Donovski]

Hi,

Since September 2007 with the release of R26 FLARE for EMC cx3 arrays, EMC supports iSCSI connectivity for HP-UX hosts. You could check EMC E-lab navigator for more details.

[Martha Mueller]

Thank you for that information. I appreciate it. How is the performance on your system now?

[Florian Heigl (new acc)]

a few tips for testing:
check if immediate_report is 1 (see scsictl man page) and raise your queue_depth a lot above the default 8.

Next:

I might sound old and grumpy now, but I think someone should really go ahead and say it aloud.
I've been working with CX storage on HP-UX since 2001 now, and used iSCSI for over 5 years.

CX series arrays and HP-UX don't go well
Hitachi, for example seems to stomp on CX both for availabilty and performance, especially since avoiding all the stupid trespassing issues (ever tuned fcp_max_requests?).
iSCSI appears fast on most platforms because of "aggressive caching", sometimes you got up to 2 minutes of outstanding writes, and I wonder if HP-UX is a bit conservative about those, this might explain the read behaviour mentioned.

Personally I'd suggest you just stay away from iSCSI for anything but lab or home networks, even 1gbit FC beats gigabit iSCSI hands down (and thats stuff so old noone still uses it!). You get better availability and the price difference isn't half as high as iSCSI vendors want to make you believe.

[Martha Mueller]

I replied to this thread an hour ago, but I don't see my response....so I'll try this again.

We have actually implemented iscsi on hp-ux 11.31 on an rx2660 to an EMC Clariion CX3-40 C. Using the Navipshere Analyzer software, we see that the response time on the iscsi luns is two to two and a half times longer than on the fibre attached luns. The metric "response time" is defined as: The average time, in milliseconds, required for one request to pass through a system component, including its waiting time.


It isn't in production yet, and we haven't finished testing things like job completion times to see if this will translate into real world problems, but I sure would have liked to have known about this issue before we invested.

[Lital Natan]

Hi, I also replied an hour ago, yet my reply does not list. So again:

I'm using iSCSI on hpux 11.23 and 11.31, on both the performance is approx 20mb/sec. (ultra slow)

My target is a linux ietd. Note that other operating systems get a rate of 60mb plus per second.

Now, pcap logs of the hp-ux iscsi show that the hpux is performing a READ for every WRITE. (although even if you count out the read, 10mb per sec is very slow for gigabit).
Needless to say, other operating systems DO NOT perform this post-write10 read.

The immediate_report is 0, but I'm unable to set it:

See output:
bash-3.2# scsictl -m ir=1 /dev/rdsk/c16t0d0
scsictl: Device ioctl (set immediate_report) error: Invalid argument

Raising the queue_depth did not change anything.

Any ideas?

[Lital Natan]

Florian, Thanks for your reply. After reading it again, I realized you probably replied to my previous post (which for some reason is not shown now...)
Anyways, my testing includes sync and, at least on linux, sync waits until the data is fully written to the disk. Doesnt that mean that I achieved 60+mb/sec -including- the actual write?
Anyways, even without the cache... transferring at 20mb/sec is ridiculous..

Further research comparing the pcap logs of hp initiator to linux initiator shows that linux writes at transfer length of 1024 (does it matter?) and hp, no matter what, at 16. That means that linux performs approx 10 writes for 100mb, and hp performs above 10000 scsi write commands over iscsi. This is probably what causes the slow down...

Why does this happen?

[Florian Heigl (new acc)]

Hi,

I'd point my finger at the immediate report setting, lacking anything else to point at.

Maybe some HP guru can shed some light on it?

I am quite sure it wasn't mentioned in the iscsi initiator docs.

A few more ideas to look at:
ndd parameters, and the window size for your transmits - this might help
raw tcp net performance test HP box <-> Linux box(i think i once used something called netperf) - this will make fingerpointing at the iSCSI initiator, as opposed to NIC or HP-UX itself, better.

My fingers are itching to do some tests myself, because I could use my iscsi-fc bridge but I don't have the time for it any time soon :(

[Bosco Tsang]

Look like the last respond was already over a year ago, has the issue been resolved yet?

We run into the similar performance issue, where we do have a 10GB uplink (we are using Itanium blade with Netapp as the central storage via 10G pipe) where the space of an iSCSI transfer is around 139MB/s while the same transfer via NFS to the same device is 894MB/s. What make up the different? We are already on the latest HPUX 11.31 with the latest version of iSCSI initiator, and it's on the delicated storage vlan without any traffic.

Bosco

[Sriram Narasimhan]

Can you disable the header/data digests and see if the performance is any better? It has been quite a while since I played with the iSCSI software initiator (I think this was in 11.23 time frame) but I thought sequential write performance (using db) was better than this (about 50MB/s for a 1K transfer size IIRC with a software Linux iSCSI offload target). Could you also vary the transfer sizes and see if the performance is any better? In the meanwhile, I will try to locate a contact for the HP-UX iSCSI software initiator.

[GaneshS]

Hi Bosco,

Will be possible to share the network diagram of your setup along with configuration details?

Thanks,
GaneshS.

[Bosco Tsang]

Attached is the captured output from iozone.

As for the infrastructure configuration diagram...

blade -> 1G -> virtual connect ->10G -> Cisco Nexus 5K -> Netapp

We are seeing lots of out of order (lost packet, varies frame size etc, account for nearly 30% of the packets being captured) when doing packet capture on before and after virtual connect (ie. on the blade and on the 10G port to N5K).

[GaneshS]

Bosco,

One more input..can you share the iSCSI operational parameter (on HP-UX>iscsiutil -p -V)

Thanks,
GaneshS.

[Bosco Tsang]

Here it is with target/address ip removed...

Operational Target Information
------------------------------

Target # 1
-----------
Target Name : iqn.1992-08.com.netapp:sn.151728088
Target Alias :
No. of Target Addresses : 2

Target Address # 1
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target Address # 2
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)
Target # 2
-----------
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 2

Target Address # 1
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target Address # 2
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Discovery Target Information
----------------------------

Target # 1
-----------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 1

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target # 2
-----------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 1

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Session information
-------------------

Session # 1
------------
Session Instance No. : 2
ISID : 0x800000000002
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728088

Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2001

Session # 2
------------
Session Instance No. : 3
ISID : 0x800000000003
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728088
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2002

Session # 3
------------
Session Instance No. : 4
ISID : 0x800000000004
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :

Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2001

Session # 4
------------
Session Instance No. : 5
ISID : 0x800000000005
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2002

[rick jones]

Lost packets are Really Bad for performance. I would suggest checking all the various queues along the way, particularly where something running at high speed meeds something running at a lower speed.