ISL performance issue between 2 site

[abouasma]

Hi All,

i'm wondering if someone can answer :-)
We setup ISL between 2 switch on different site distanced by 5KM. So i'm talking about on Fabric, The Configuration is as follow:
2x2GB/s Port 30/31

More information in attached file saninput

As you can see the max performance pert port reached is 60MB, so Total of 120GB.

My question is, should i expect more as we have a trunk of 4GB/s bandwith.
Then were should be the issue, BB-credit ?
The distance is less tahn 10KM, so ??

[IBaltay]

Hi, sure that the trunk is the best option but you need to have trunking license:

Look at this discussion:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1287709

and here is the major part of the needed info:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00403562/c00403562.pdf?jumpid=reg_R1002_USEN

[abouasma]

Hi IBtalay,

In fact we have ISL trunk of 2 x 2GB/s as i mentionnned in my first message and we have installed ISL trunking installed on all switch, we can see that IO are well balanced through port.
But my question was regarding the throuput of 60MB/s pert port. Is that OK for a 2GB port? I would expect a performance reaching 200MB per port, so a Total of 400MB. No ???

Thanks

[IBaltay]

Hi,
Yes 2 ISL ports should give 400MB/s.
Look at here in the Exploring Brocade trunking doc:
http://www.hds.com/assets/pdf/brocade_wp_trunking.pdf

and
the 6.10 troubleshooting/diagnostic guide p 61-62:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01482877/c01482877.pdf?jumpid=reg_R1002_USEN

for some ideas to troubleshoot/diagnose your ISL/trunked links.

also the ISL oversubscription could be an issue here, from the page 101 in the SAN HA document:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00196284/c00196284.pdf?jumpid=reg_R1002_USEN

[abouasma]

HI Ibaltay

I read the document, and unfortunatly this didn't answer my question. I'm not running oversubscrtiption and i don't think having isew with deskew.
I'm now thinking about the nature of IO, as we try to do mirroring between 2 storage, i will try with dd to see if i get the same performance.

Best regards

[PP BIJU KRISHNAN]

What's in between the 2 sites??

Do you see errors on the ports or in errdump or errshow??

Can you provide o/p of

portstatsshow
portshow

for the 2 ISL ports.

Also porterrshow for the entire switch.

[abouasma]

Hi PP BIJU,

No error in errordump, or errshow
The rest of the command can be found in attched file, strange er_bad_os value in portstatshow on port31,

Thank's for your help

[abouasma]

Hi PP BIJU,

Could you tell me what's mean

"tim_txcrd_z 527422 Time BB credit zero"

Trunking License, and extended Fabric license are installed

B.R

[IBaltay]

Hi,
wiki explanation:
http://en.wikipedia.org/wiki/Buffer_credits

look at this threads too...
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1232958322424+28353475&threadId=1135333

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=629357

sun forum discussion:
http://forums.sun.com/thread.jspa?threadID=5208613

[RUET]

hello all
i also suspect something wrong witha interlink ISL:
What do you think about these values collected :
stat_wtx 673414948 4-byte words transmitted
stat_wrx 2953345 4-byte words received
stat_ftx 2003234271 Frames transmitted
stat_frx 2225058392 Frames received
stat_c2_frx 0 Class 2 frames received
stat_c3_frx 2219943598 Class 3 frames received
stat_lc_rx 2492784 Link control frames received
stat_mc_rx 0 Multicast frames received
stat_mc_to 0 Multicast timeouts
stat_mc_tx 0 Multicast frames transmitted
tim_rdy_pri 189170 Time R_RDY high priority
tim_txcrd_z 2922932550 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 6571232 0 2916361318 0
tim_txcrd_z_vc 4- 7: 0 0 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0
er_enc_in 1 Encoding errors inside of frames
er_crc 1493 Frames with CRC errors
er_trunc 0 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 0 Frames with bad end-of-frame
er_enc_out 106854 Encoding error outside of frames
er_bad_os 45408680 Invalid ordered set
Thanks in advance for any comments

[Johan Guldmyr]

I would suggest that maybe you can open your own post?

Stats like that can accumulate for a long time.
It's more interesting to see where they are increasing.

clear stats.
wait.
then look at the numbers.

or use dcfm or something to make graphs of your stats..

[RUET]

Yes you are rigth, here is new delta value for a 6 days period, where stats have been cleared before this period:

Port 4
Status Online
Type E
Speed 2G
Tx Frames 752247510
Rx Frames 950434936
LLI 531851
LLI to frame ratio 0.0312%
Encoding errors inside frame 39100
CRC errors 13328
Invalid CRC 8027
CRC errors g_eof 8004
Invalid word 513101
Frame to short 35
Frame to long 0
Bad end of frame 5324
Encoding errors outside frame 473900
Discarded Class 3 frame 1698

This is more readable !!!
I suspect the ISL link, or GBIC , or fibers .

Do you have any ideas ?
Thanks you

[Johan Guldmyr]

Often, "enc out" errors alone means cable problems.
"enc out" and "crc err" often means GBIC/SFP problems.

[Amar_Joshi]

Abouasma,
One more thing to check what is the connection between 2 sites? Are you leasing or it's a dark fiber? In some cases provider provides the FCIP sort of connections and bottleneck resides in their routes. So it will be good to check that as well.

[RUET]

we are leasing a solution to a company, based on MRV200E boxes, with 2 links (one for each fabric) with a bandwith of 1Gb/s.
We suspect or the cable or SFP, or the leaed connection ...
Lenght is supposed to be 12km ..
do you think it can be a buffer to buffer credit problem as we don't have extended fabric on our switches ?

[Johan Guldmyr]

For some DWM/FCIP solutions you do not need extended fabric.