Performance problem w/FC2684 HBA on DS10

[Thomas Simpson]

We have installed new FC2684 HBA adapters on multiple DS10 systems. The (SAN disk) performance is very slow as viewed from the switch (about 500KB/s). I've looked over the installation and release notes and see nothing helpful. I've installed all of the current patches for OpenVMS 7.3-2 and upgraded the HBA and DS10 firmware (7.2).

A few of the DS10 systems already have older older HBAs installed and work fine, just the new ones are slow.

Any ideas what might be wrong or what to check next? Are there known configuration issues with this hardware combination? I did not see anything in my searches that would indicate a problem.

Regards,
Tom

[Volker Halle]

Tom,

you are refering to: FCA2684 DS-KGPSA-EA LP9802, right ?

There seems to be a mandatory FW version of 1.00x8 - check with SDA> FC SHOW DEV FGA

You can check for FC-related problems with:

SDA> FC SHOW STDT/ALL

You can look at the FC ring buffer with

SDA> FC SHOW RING

to look for any unusual errors/timeouts.

Volker.

[Thomas Simpson]

Volker,

Thanks for the quick (and helpful, as usual) response!

I believe the FCA2684 is not the same as DS-KGPSA-EA. As best I can tell, the part number is DS-A5134-AA. I updated firmware with the latest version for this HBA per the HP web page.

I do see some bad status entries in the ring buffer...

Here is what SDA says:

SDA> fc show dev fgc
FGC0: operational firmware revision TS1.91X6
port_name(adapter_id) = 1000-0000-C955-D075, node_name(host_id) = 2000-0000-C955-D075

SDA> FC SHOW STDT/ALL

PGC0 SPDT 816C5100 STDTs
------------------------
| STDT PRLI Port Dev Cred Appr | Act Cmd Cnf Rst PRLO Cls | QF Tgt Ill Seq
STDT FC-LA Port Name | Stat Stat I/Os I/Os I/Os I/Os | Sus Sus Pnd Act Pnd Pau | Seen Rsts Frms Tmo
-------- ----- ------------------- + ---- ---- ---- ---- ---- ---- + --- --- --- --- --- --- + ---- ---- ---- ----
816CAB40 00003 5000.1FE1.5004.1E99 | 0001 0001 0000 0001 0000 0000 | 000 000 000 000 000 000 | 0000 0000 0000 0000
816D09C0 00004 5000.1FE1.5004.1E9D | 0001 0001 0000 0000 0000 0000 | 000 000 000 000 000 000 | 0000 0000 0000 0000

SDA> FC SHOW RING

FGC0: Ring Buffer Trace Information from RBD 815754D0
-----------------------------------------------------
Date Time Event ID Data Buff [7] Buff [6] Buff [5] Buff [4] Buff [3] Buff [2] Buff [1] Buff [0] RBE
---- ---- -------- ---- -------- -------- -------- -------- -------- -------- -------- -------- --------
6-SEP 15:28:47.42 Write IOCB 00010019 1D229806 00040005 00000000 00008000 00000000 00000000 811B0A10 40000048 815BCF00
6-SEP 15:28:47.42 Bad IOCB Sts 00000014 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 815BCED0
6-SEP 15:28:47.42 Read IOCB 00010018 10229936 00460002 00000000 0000000D 0008EC56 00000000 8149B070 40000048 815BCEA0
6-SEP 15:28:47.42 Read IOCB 00010017 18A29906 00490002 00000000 00008000 0008EC55 00000000 8149B070 40000054 815BCE70
6-SEP 15:28:47.42 Read IOCB 00010016 18A29906 00440002 00000000 00008000 0008EC54 00000000 8149B070 40000054 815BCE40
6-SEP 15:28:47.42 Read IOCB 00010015 18A29906 00450002 00000000 00008000 0008EC53 00000000 8149B070 40000054 815BCE10
6-SEP 15:28:47.42 Read IOCB 00010014 18A29906 00470002 00000000 00008000 0008EC52 00000000 8149B070 40000054 815BCDE0
6-SEP 15:28:47.42 Read IOCB 00010013 18A29906 00480002 00000000 00008000 0008EC51 00000000 8149B070 40000054 815BCDB0
6-SEP 15:28:47.42 Read IOCB 00010012 18A29906 00430002 00000000 00008000 0008EC50 00000000 8149B070 40000054 815BCD80
6-SEP 15:28:47.41 Read IOCB 00010011 18A29906 00420002 00000000 00008000 0008EC4F 00000000 8149B070 40000054 815BCD50
6-SEP 15:28:47.41 Read IOCB 00010010 18A29906 00410002 00000000 00008000 0008EC4E 00000000 8149B070 40000054 815BCD20
6-SEP 15:28:47.41 Read IOCB 0001000F 18A29906 00400002 00000000 00008000 0008EC4D 00000000 8149B070 40000048 815BCCF0
6-SEP 15:28:47.41 Read IOCB 0001000E 18A29906 003E0002 00000000 00008000 0008EC4C 00000000 8149B070 40000048 815BCCC0
6-SEP 15:28:47.41 Read IOCB 0001000D 1DA29906 003F0004 00000000 00008000 00000000 00000000 811B0A10 4000003C 815BCC90
6-SEP 15:28:44.42 Write IOCB 0001000D 1D229806 00040004 00000000 00008000 00000000 00000000 811B0A10 4000003C 815BCC60
6-SEP 15:28:44.42 Bad IOCB Sts 00000014 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 815BCC30
6-SEP 15:28:44.42 Read IOCB 0001000C 10229936 003D0002 00000000 0000000D 0008EC23 00000000 8149B070 4000003C 815BCC00


FGC0: Ring Buffer Trace Information from RBD 815754D0
-----------------------------------------------------
Date Time Event ID Data Buff [7] Buff [6] Buff [5] Buff [4] Buff [3] Buff [2] Buff [1] Buff [0] RBE
---- ---- -------- ---- -------- -------- -------- -------- -------- -------- -------- -------- --------
6-SEP 15:28:44.42 Read IOCB 0001000B 18A29906 003C0002 00000000 00008000 0008EC21 00000000 8149B070 4000003C 815BCBD0
6-SEP 15:28:44.42 Read IOCB 0001000A 18A29906 00390002 00000000 00008000 0008EC1F 00000000 8149B070 4000003C 815BCBA0
6-SEP 15:28:44.42 Read IOCB 00010009 18A29906 003B0002 00000000 00008000 0008EC1E 00000000 8149B070 40000054 815BCB70
6-SEP 15:28:44.42 Read IOCB 00010008 18A29906 003A0002 00000000 00008000 0008EC1C 00000000 8149B070 40000054 815BCB40
6-SEP 15:28:44.42 Read IOCB 00010007 18A29906 00380002 00000000 00008000 0008EC19 00000000 8149B070 40000048 815BCB10
6-SEP 15:28:44.42 Read IOCB 00010006 18A29906 00370002 00000000 00008000 0008EC17 00000000 8149B070 40000054 815BCAE0
6-SEP 15:28:44.41 Read IOCB 00010005 18A29906 00350002 00000000 00008000 0008EC10 00000000 8149B070 40000054 815BCAB0
6-SEP 15:28:44.41 Read IOCB 00010004 18A29906 00340002 00000000 00008000 0008EC07 00000000 8149B070 40000054 815BCA80
6-SEP 15:28:44.41 Read IOCB 00010003 1DA29906 00310005 00000000 00008000 00000000 00000000 811B0A10 40000054 815BCA50
6-SEP 15:28:41.69 Bad IOCB Sts 00000014 10001132 00360014 00000000 00000005 00000000 00000000 00000000 00000000 815BCA20
6-SEP 15:28:41.69 Read IOCB 00000003 10001132 00360014 00000000 00000005 00000000 00000000 00000000 00000000 815BC9F0
6-SEP 15:28:41.68 Write IOCB 00000003 00008606 0000000B 00000000 00000000 00000000 00000000 812C2850 00000830 815BC9C0
6-SEP 15:28:41.68 Write IOCB 00000002 00001102 00360014 00000000 00000000 00000000 00000000 00000000 00000000 815BC990
6-SEP 15:28:41.68 Read IOCB 00000002 00028106 00360003 03200080 00000000 00000000 00000000 812C2850 0000001C 815BC960
6-SEP 15:28:41.62 Read IOCB 00000001 00028306 00360014 0220000C 00000000 00000000 00000000 8117CD90 4000000C 815BC930
6-SEP 15:28:41.62 Write IOCB 00000001 00028206 00030014 0220000C 00000000 00000000 00000000 8117CD90 4000000C 815BC900
6-SEP 15:28:41.43 Write IOCB 00010003 1D229806 00040005 00000000 00008000 00000000 00000000 811B0A10 40000054 815BC8D0

[Volker Halle]

Tom,

FCA2684 Single Channel DS-A5132-AA
FCA2684DC Dual Channel DS-A5134-AA

I do not see the Dual Channel adapter listed as a supported option for the DS10. It is listed as supported (LP10000DC) for a DS25.

There is also a long list of supported FC switches associated with each FC adapter.

You do have (at least) another 2 FC adapters in these machines. Which types ? Do they work correctly ?

Volker.

[Thomas Simpson]

We have a KGPSA-CX HBA in the same system that works just fine. The literature said that the HBA was backward compatible so we picked the dual channel model to deal with the DS10 slot restrictions.

Further testing indicates that the problem seems to be a write (or transmit) problem.

After we finally were able to complete a backup to a disk connected via this adapter, we were able to test the read speed. It looks pretty good, showing an I/O rate of 1400-2300 I/O per second (using MONITOR) from a raid5 EVA disk.

We are fairly certain the problem is not the EVA because we see the same problem writing to tapes, but we are going to zone in a different disk controller (HSG80) and test again to eliminate the possibility of an EVA problem.

Regards,
Tom

[Thomas Simpson]

Further testing indicates that the new fiber channel HBA appears to work fine when writing to an HSG80 disk. A sustained I/O rate of approximately 500 I/O per second was observed while performing a backup to the disk.

The same test using the EVA raid 5 disk produced an I/O rate of about 20. Definitely the not the performance I'd expect...even from a raid5 configuration.

So now it's on to the EVA and ESL tape library issues...

[Volker Halle]

Tom,

this is not a 'performance issue', but something goes wrong and is retried only after a timeout of 3 seconds. Look at the timing of the Bad IOCB sts message and the next read/write command. You'll see gaps of about 3 seconds (the ring buffer is shown with most recent time listed first).

Volker.

[Hein van den Heuvel]

>> The same test using the EVA raid 5 disk produced an I/O rate of about 20. Definitely the not the performance I'd expect...even from a raid5 configuration.

What is the CLUSTERSIZE on the disk?

There is a known issue with slow writes to EVA raid-5 when transfers do NOT start at Logical block numbers with mutliples of 4.
In this 'odd' case the EVA failed to see it could avoid the reads when writing large chunks. Those read are needed small block writes and will hurt a bunch (though 20 IO/sec seems crazy).

Other (Unix) OSes tend to use 8KB aligned disk allocation algoritmes and did not notice this problem with the EVA.

The first step to start out aligned is to have a cluster size with being a multiple of 4. The next step is to keep the IOs aligned by having IO sizes being multiples of 4.

That's why COPY with 8.2 defaults to 124, not the 127 max and it got a /BLOCK_SIZE switch.

hth,
Hein van den Heuvel
HvdH Performance Consulting

[Thomas Simpson]

The cluster size for the test disk is 1808, which can be evenly divided by 4. However, the cluster size of the source disk was 69. Could that cause the problem you were referring to?

The 20 I/O per second figure was a peak...the average rate was more like 1-3 I/O sec.

We tried building a smaller raid5 disk on the EVA for testing. We got the same results. The default cluster size was used.

I'm not sure about setting the I/O size. Can you explain? I know you can set the Backup block size... Is that what you were talking about? I do not see a way to set the transfer size on the EVA disk as you can on the HSG80 disks.

We have opened a trouble ticket with HP. Storageworks support is looking at it now.

Regards,
Tom

[Hein van den Heuvel]

The effect I was referring to is less than 2x, an you are seeing 100x too slow.

So I defer to Volkers observation on the 3 second delays. For some reason some IO gets stuck, the system waits 3 seconds and then it works. Some wrong path through a switch, something.

Good to hear you started a support call.

Hein.