iscsi initiator poor performance

[Dejan Donovski]

We have tested HP-UX software iSCSI initiator with EMC cx3-20c target. Although HP-UX iSCSI initiator is not supported it works except for the performance problem. We were not able to achieve more then 20MB/s on a iscsi vxfs file system. This is very poor. I could not find any docs regarding iscsi initiator optimization and the only configurable iscsi parameters are authentication related.
The iscsi stats are attached, however i am not able to interpret them. The only suspicious parameter is I/O underflows.

Any advices

[Britto Sidhan]

Hi

I don't know how much the following article will be useful for you. Just check it:

hxxp://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/9687/30571/01410738.pdf

For this you need an userid and password in ieeexplore.

Thanks and Regards
Britto Sidhan

[rick jones]

I would start checking TCP and link-level stats. Netstat -s -p tcp snapshots before and after your transfer, then run through ftp://ftp.cup.hp.com/dist/networking/tools/beforeafter.c

Look for things like retransmissions, and retransmission timeouts and out of order segments.

Do the same thing with lanadmin -g mibstats before and after and look for errors, discards and _late_ collisions.

Also, what sort of system is this, and what is the per-CPU utilization of each CPU while the transfer is taking place? I'm guessing this is a gigabit/s ethernet, but any idea how many I/Os can be outstanding at one time, and of what size?

What is the TCP window size being used - you want to get a packet trace starting from when the target is first mounted so you can see the window scaling option if it is present.

[Martha Mueller]

Now it is a year later, and our company is considering installing an EMC clariion that would be used by two servers: an rx3600 and an rx4640, both running HP-UX 11.23 (that's 11i V2) over iSCSI.

Can anyone comment on this configuration? Found this HP web site, "The following third-party storage devices have completed independent interoperability validation under the guidance of HP and have been qualified as interoperable in HP-UX environments."

http://www.hp.com/products1/serverconnectivity/mass_storage_devices.html#EMC

But it appears that only HP-UX 11iV3 is listed.

I haven't found anything that discusses integrity servers running HP-UX iSCSI with Auto Port Aggregation. Would aggregating two ethernet ports work well with iSCSI?

Has anyone tried this arrangement, am I doomed to have the same performance issues that Dejan had last year? Will there still be performance issues (or any other issues) if we attempt to use iSCSI even without aggregating ports for data? We'll keep network traffic separate from data traffic.

Thanks

[Dejan Donovski]

Hi,

Since September 2007 with the release of R26 FLARE for EMC cx3 arrays, EMC supports iSCSI connectivity for HP-UX hosts. You could check EMC E-lab navigator for more details.

[Martha Mueller]

Thank you for that information. I appreciate it. How is the performance on your system now?

[Florian Heigl (new acc)]

a few tips for testing:
check if immediate_report is 1 (see scsictl man page) and raise your queue_depth a lot above the default 8.

Next:

I might sound old and grumpy now, but I think someone should really go ahead and say it aloud.
I've been working with CX storage on HP-UX since 2001 now, and used iSCSI for over 5 years.

CX series arrays and HP-UX don't go well
Hitachi, for example seems to stomp on CX both for availabilty and performance, especially since avoiding all the stupid trespassing issues (ever tuned fcp_max_requests?).
iSCSI appears fast on most platforms because of "aggressive caching", sometimes you got up to 2 minutes of outstanding writes, and I wonder if HP-UX is a bit conservative about those, this might explain the read behaviour mentioned.

Personally I'd suggest you just stay away from iSCSI for anything but lab or home networks, even 1gbit FC beats gigabit iSCSI hands down (and thats stuff so old noone still uses it!). You get better availability and the price difference isn't half as high as iSCSI vendors want to make you believe.

[Martha Mueller]

I replied to this thread an hour ago, but I don't see my response....so I'll try this again.

We have actually implemented iscsi on hp-ux 11.31 on an rx2660 to an EMC Clariion CX3-40 C. Using the Navipshere Analyzer software, we see that the response time on the iscsi luns is two to two and a half times longer than on the fibre attached luns. The metric "response time" is defined as: The average time, in milliseconds, required for one request to pass through a system component, including its waiting time.


It isn't in production yet, and we haven't finished testing things like job completion times to see if this will translate into real world problems, but I sure would have liked to have known about this issue before we invested.

[Lital Natan]

Hi, I also replied an hour ago, yet my reply does not list. So again:

I'm using iSCSI on hpux 11.23 and 11.31, on both the performance is approx 20mb/sec. (ultra slow)

My target is a linux ietd. Note that other operating systems get a rate of 60mb plus per second.

Now, pcap logs of the hp-ux iscsi show that the hpux is performing a READ for every WRITE. (although even if you count out the read, 10mb per sec is very slow for gigabit).
Needless to say, other operating systems DO NOT perform this post-write10 read.

The immediate_report is 0, but I'm unable to set it:

See output:
bash-3.2# scsictl -m ir=1 /dev/rdsk/c16t0d0
scsictl: Device ioctl (set immediate_report) error: Invalid argument

Raising the queue_depth did not change anything.

Any ideas?

[Lital Natan]

Florian, Thanks for your reply. After reading it again, I realized you probably replied to my previous post (which for some reason is not shown now...)
Anyways, my testing includes sync and, at least on linux, sync waits until the data is fully written to the disk. Doesnt that mean that I achieved 60+mb/sec -including- the actual write?
Anyways, even without the cache... transferring at 20mb/sec is ridiculous..

Further research comparing the pcap logs of hp initiator to linux initiator shows that linux writes at transfer length of 1024 (does it matter?) and hp, no matter what, at 16. That means that linux performs approx 10 writes for 100mb, and hp performs above 10000 scsi write commands over iscsi. This is probably what causes the slow down...

Why does this happen?