iscsi initiator poor performance

[Florian Heigl (new acc)]

Hi,

I'd point my finger at the immediate report setting, lacking anything else to point at.

Maybe some HP guru can shed some light on it?

I am quite sure it wasn't mentioned in the iscsi initiator docs.

A few more ideas to look at:
ndd parameters, and the window size for your transmits - this might help
raw tcp net performance test HP box <-> Linux box(i think i once used something called netperf) - this will make fingerpointing at the iSCSI initiator, as opposed to NIC or HP-UX itself, better.

My fingers are itching to do some tests myself, because I could use my iscsi-fc bridge but I don't have the time for it any time soon :(

[Bosco Tsang]

Look like the last respond was already over a year ago, has the issue been resolved yet?

We run into the similar performance issue, where we do have a 10GB uplink (we are using Itanium blade with Netapp as the central storage via 10G pipe) where the space of an iSCSI transfer is around 139MB/s while the same transfer via NFS to the same device is 894MB/s. What make up the different? We are already on the latest HPUX 11.31 with the latest version of iSCSI initiator, and it's on the delicated storage vlan without any traffic.

Bosco

[Sriram Narasimhan]

Can you disable the header/data digests and see if the performance is any better? It has been quite a while since I played with the iSCSI software initiator (I think this was in 11.23 time frame) but I thought sequential write performance (using db) was better than this (about 50MB/s for a 1K transfer size IIRC with a software Linux iSCSI offload target). Could you also vary the transfer sizes and see if the performance is any better? In the meanwhile, I will try to locate a contact for the HP-UX iSCSI software initiator.

[GaneshS]

Hi Bosco,

Will be possible to share the network diagram of your setup along with configuration details?

Thanks,
GaneshS.

[Bosco Tsang]

Attached is the captured output from iozone.

As for the infrastructure configuration diagram...

blade -> 1G -> virtual connect ->10G -> Cisco Nexus 5K -> Netapp

We are seeing lots of out of order (lost packet, varies frame size etc, account for nearly 30% of the packets being captured) when doing packet capture on before and after virtual connect (ie. on the blade and on the 10G port to N5K).

[GaneshS]

Bosco,

One more input..can you share the iSCSI operational parameter (on HP-UX>iscsiutil -p -V)

Thanks,
GaneshS.

[Bosco Tsang]

Here it is with target/address ip removed...

Operational Target Information
------------------------------

Target # 1
-----------
Target Name : iqn.1992-08.com.netapp:sn.151728088
Target Alias :
No. of Target Addresses : 2

Target Address # 1
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target Address # 2
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)
Target # 2
-----------
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 2

Target Address # 1
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target Address # 2
-------------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Discovery Target Information
----------------------------

Target # 1
-----------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 1

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Target # 2
-----------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 1

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Session information
-------------------

Session # 1
------------
Session Instance No. : 2
ISID : 0x800000000002
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728088

Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2001

Session # 2
------------
Session Instance No. : 3
ISID : 0x800000000003
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728088
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2002

Session # 3
------------
Session Instance No. : 4
ISID : 0x800000000004
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2001

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :

Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2001

Session # 4
------------
Session Instance No. : 5
ISID : 0x800000000005
Session State : Online
Target Name : iqn.1992-08.com.netapp:sn.151728129
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : *
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 2002

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)

Login Keys:
-----------

MaxConnections : 1
AuthMethod Proposed : None
AuthMethod Chosen : None
MaxRecvDataSegmentLength : 16384 bytes
MaxBurstLength : 65536 bytes
FirstBurstLength : 65536 bytes
DefaultTime2Wait : 0 seconds
DefaultTime2Retain : 20 seconds
MaxOutstandingR2T : 1
ErrorRecoveryLevel : 0 (Session failure recovery)
InitialR2T : No
ImmediateData : Yes
DataPDUInOrder : Yes
DataSequenceInOrder : Yes
OFMarker : No
IFMarker : No
OFMarkInt : 0
IFMarkInt : 0
DataDigest Proposed : None,CRC32C
DataDigest Chosen : None
HeaderDigest Proposed : None,CRC32C
HeaderDigest Chosen : None
TargetAddress : *:3260,2002

[rick jones]

Lost packets are Really Bad for performance. I would suggest checking all the various queues along the way, particularly where something running at high speed meeds something running at a lower speed.