Poor performance with EVA4400 and OpenVMS 8.4 using QLogic ISP2422

[Thomas Sonnentag]

Hello,

our new environment with FC-Disks in a EVA4400 shows a throughput of about 40 MByte/s
(using copy, backup and Oracle/Rdb database access). I think that is to less and there is a problem somewhere.
For the test a simple connection form the VMS-System to the switch and form the switch to the EVA is used. Speed is checked with evaperf and the monitor utility on the Switch. There are no errors on the Fibre Channel.

Is it possible that something goes wrong concerning the Fibre Channel IO ?

[labadie_1]

Hello

use the FC Sda extension

$ ana/sys
fc perf ...

to do some measurements

SDA> fc

FibreChannel SDA Extension output is best viewed in at least 132 columns

Supported commands:

FC ADDRESS_LIST
FC KPB
FC NAME_LIST
FC PERFORMANCE [/CSV] [/NOCOMPRESS] [/RSCC | /SYSTIME] [/DISK] [/TAPE] [/ALL
] [/CLEAR] [device-name]
FC PROBE_LIST
FC QUEUES [/LIST]
FC SCDT
FC SET DEVICE [device-name]
FC SET ERL /SIZE=entry-count [/ALL]
FC SET FILTER [/ASC[=list]] [/COMMAND[=list]] [/ESTATUS[=list]]
[/FCPSTATUS[=list]] [/WWID[=list-quoted-strings]]
[/MATCH={AND | OR | NOR}] [/APPEND] [/ALL]
FC SET RING_BUFFER /SIZE=entry-count
[/FCP|/SLOW|/ERROR|/INTERRUPTS|/MBX|/IOCB] [/ALL]
FC SET WTID /WWID=quoted-string [/CAP=cap-value] [/[NO]WAIT]
Press RETURN for more.
SDA>

[Thomas Sonnentag]

This is what I get writing 100 * 1 MB:

$1$dga3005 (write)
Using EXE$GQ_SYSTIME to calculate the I/O time
accumulated write time = 2115017us
writes = 12809
total blocks = 412809

LBC <2ms <8ms <16ms <32ms <128ms
=== ======== ======== ======== ======== ========
1 3208 - - 1 - 3209
32 6378 16 4 - 2 6400
64 3200 - - - - 3200

12786 16 4 1 2 12809

The Performance is about 40 MByte/s with our testprogram. If the disk is VMS-Shadowed the spead will be nearly the half...

[Thomas Sonnentag]

Same program writing to a shadowed disk:

FibreChannel Device Performance Data
------------------------------------

$1$dga3100 (write)
Using EXE$GQ_SYSTIME to calculate the I/O time
accumulated write time = 3552016us
writes = 15121
total blocks = 414962

LBC <2ms <4ms <8ms <16ms <32ms <64ms
=== ======== ======== ======== ======== ======== ========
1 5360 - - - 2 - 5362
16 188 - 1 - - - 189
32 6298 1 77 17 10 1 6404
64 3164 1 1 - - - 3166

15010 2 79 17 12 1 15121


FibreChannel Device Performance Data
------------------------------------

$1$dga4100 (write)
Using EXE$GQ_SYSTIME to calculate the I/O time
accumulated write time = 3181009us
writes = 15121
total blocks = 414962

LBC <2ms <4ms <8ms <16ms <32ms <64ms
=== ======== ======== ======== ======== ======== ========
1 5361 - - - - 1 5362
16 188 - 1 - - - 189
32 6299 10 74 15 3 3 6404
64 3165 - - 1 - - 3166

15013 10 75 16 3 4 15121

[Thomas Sonnentag]

I did further test with the local SCSI-Disks on our rx7640.
The write speed is about 1.2 MByte/sec.
This leads me to the point that there is no specific problem with the FC, I think there is a general disk I/O problem.

[Thomas Sonnentag]

Is it possible that VMS on Itanium does "Interrupt Coalescing for I/O Performance Gains" ? If yes can I turn it off ?
I read about this and tried it on our Alpha systems, but the application performance was quite bad by changing the default values !

[Jim_McKinney]

Regarding FC interrupt coalescing, see the output of

$ mc sys$etc:fc$cp

Early on, any interrupt coalescing was "off" by default - that may have changed in recent versions of VMS.

[Thomas Sonnentag]

fc$cp does not work on Itanium...

[Hein van den Heuvel]

Hi Thomas,

Good work, trying to capture generically observed complaints with a concrete and reproducible test.
Please help me understand the test program/result matrix a little better.

I suspect that you use $QIO into a pre-allocated file, thus avoiding any (RMS) buffering and file allocation variables. Correct?

And the source data for the write is simple, predictable, pre-filled (or not-filled) memory buffers right? It is not freshly read data is it?

From the matrix it looks like you used a mix of IO sizes in the same run. Correct?

What about the write pattern? Simple linear?
I suspect you are using a using a random picked size from a list following a target distribution, to a random target in range. Correct?

I like the result presentation in general, bit given the observerd time ranges, the < 2ms seems too coarse.
Perhaps also display an average time with a decimal fraction for each LBC group?

Have you compared with the XFC enabled for the target file(s)?
$ SET FILE /CACHING=NO_CACHING ...

Is this new with 8.4, or observed under 8.3 already but shown under 8.4 to make clear you are using the 'latest and greatest' ?

Now < 2ms is less than the rotational delay for a fast disk: 15Krpm is 60,000 / 15,000 = 4 ms per revolution. So you are apparently measuring the Write-Back Control Caching performance, not the spindle performance.
That's fine but needs to be recognized.

An aggregate write of 40 MB is interestingly close to per-logical-unit write cache sizes typically deployed by controller caches.
How many spindles behind the unit to 'suck up / sink ' the data from the cache?

What platform? Itanium/Alpha? RX7640?

Is this 1gb fibre, or more?

For 1gb, that 40MB makes a good dent into the max throughput.

Are the shadowed IOs going over a single FC or two?

Now sure what to make of the SCSI io number. Those seem ridiculously low.

Hope this helps some,
Hein van den Heuvel
HvdH Performance Consulting

[Thomas Sonnentag]

Hello Hein,

I don't know exactly what the testprogram does. As far as I know it allocates 1MB of memory and writes it 100 times to disk.

The testprogram helps me to get a qick first impression if I change something on the system. The result from this program is quite close to writes from backup image savesets, disk copies, copies, and oracle Rdb accesses.

Our Environment are severel clusters based on rx7640 with OpenVMS 8.4 and latest patches. All these systems are connected with two 4 GB FC-Adapter through a Brocade switch to the EVA 4400.

I'm currently checking the settings on the FC-Adapter (EFI -> Fibre Channel Driver).