ISL problem beetween 2 sites

[RUET]

hello a new thread a weel known subject, but as I don't find the solution , may be you can give me some ideas.
I have a SAN (2 fabric) on two sites. Two 1Gb links (one per fabric) are leased. I experienced some performance problem, and I don't know where I have to find ..
May be hardware SFP/fibers ..
may be the interlink leased fiber
may be some bad config on ISL .. (buffer credit, ISL Ready mode that is not set in my switches ..)

i made a portstat64show and I get this :
tat64_wtx 194 top_int : 4-byte words transmitted
2748752667 bottom_int : 4-byte words transmitted
stat64_wrx 188 top_int : 4-byte words received
1940498789 bottom_int : 4-byte words received
stat64_ftx 0 top_int : Frames transmitted
2299469278 bottom_int : Frames transmitted
stat64_frx 0 top_int : Frames received
2298266388 bottom_int : Frames received
stat64_c2_frx0 top_int : Class 2 frames received
0 bottom_int : Class 2 frames received
stat64_c3_frx0 top_int : Class 3 frames received
2298166835 bottom_int : Class 3 frames received
stat64_lc_rx0 top_int : Link control frames received
71438 bottom_int : Link control frames received
stat64_mc_rx0 top_int : Multicast frames received
0 bottom_int : Multicast frames received
stat64_mc_to0 top_int : Multicast timeouts
0 bottom_int : Multicast timeouts
stat64_mc_tx0 top_int : Multicast frames transmitted
0 bottom_int : Multicast frames transmitted
tim64_rdy_pri0 top_int : Time R_RDY high priority
10479 bottom_int : Time R_RDY high priority
tim64_txcrd_z0 top_int : Time BB_credit zero
4045036478 bottom_int : Time BB_credit zero
er64_enc_in 0 top_int : Encoding errors inside of frames
90909 bottom_int : Encoding errors inside of frames
er64_crc 0 top_int : Frames with CRC errors
19252 bottom_int : Frames with CRC errors
er64_trunc 0 top_int : Frames shorter than minimum
71 bottom_int : Frames shorter than minimum
er64_toolong0 top_int : Frames longer than maximum
0 bottom_int : Frames longer than maximum
er_bad_eof 0 top_int : Frames with bad end-of-frame
12632 bottom_int : Frames with bad end-of-frame
er64_enc_out0 top_int : Encoding error outside of frames
1347042 bottom_int : Encoding error outside of frames
er64_disc_c30 top_int : Class 3 frames discarded
9456 bottom_int : Class 3 frames discarded
stat64_rateTxFrame9670 Tx frame rate (fr/sec)
stat64_rateRxFrame5120 Rx frame rate (fr/sec)
stat64_rateTxPeakFrame53430 Tx peak frame rate (fr/sec)
stat64_rateRxPeakFrame57166 Rx peak frame rate (fr/sec)
stat64_rateTxWord4144304 Tx Word rate (words/sec)
stat64_rateRxWord1832537 Rx Word rate (words/sec)
stat64_rateTxPeakWord26874975 Tx peak Word rate (words/sec)
stat64_rateRxPeakWord27531128 Rx peak Word rate (words/sec)
stat64_PRJTFrames0 top_int : 4-byte words transmitted
0 bottom_int : 4-byte words transmitted
stat64_PBSYFrames0 top_int : 4-byte words transmitted
0 bottom_int : 4-byte words transmitted
stat64_inputBuffersFull0 top_int : 4-byte words transmitted
0 bottom_int : 4-byte words transmitted
stat64_rxClass1Frames0 top_int : 4-byte words transmitted
0 bottom_int : 4-byte words transmitted

I made a delta for a six days period, and the main resume of the portstatshow command was :
Type E
Speed 2G
Tx Frames 752247510
Rx Frames 950434936
LLI 531851
LLI to frame ratio 0.0312%
Encoding errors inside frame 39100
CRC errors 13328
Invalid CRC 8027
CRC errors g_eof 8004
Invalid word 513101
Frame to short 35
Frame to long 0
Bad end of frame 5324
Encoding errors outside frame 473900
Discarded Class 3 frame 1698

Any ideas
thanks in advance
Pat

[Jan Soska]

Hello,
some more detail on models, fabric os, etc?

jan

[Malcolm Wade]

A rule of thumb I use which came from my Brocade 2Gb switch training is:

"Statistically, enc_out errors alone imply cable problems, the enc_out and crc_err combination implies GBIC/SFP problem"

If this is a Brocade switch, pls post a porterrshow but I'd start by changing the GBIC/SFP if you have a spare of same type somewhere.

[RUET]

Here are some infos:
ortshow 4
portName:
portHealth: No Fabric Watch License
Failed to get authentication information.
portDisableReason: None
portCFlags: 0x1
portFlags: 0x1000490b PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
portType: 18.0
portState: 1Online
Protocol: FC
portPhys: 6In_Sync portScn: 16E_Port
port generation number: 38
state transition count: 3

portId: 040400
portIfId: 43020012
portWwn: 20:04:00:05:1e:a8:73:e0
portWwn of device(s) connected:
20:07:00:05:1e:04:a3:1e
Distance: standard <= 10km
portSpeed: 2Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts: 0 Link_failure: 0 Frjt: 0
Unknown: 0 Loss_of_sync: 0 Fbsy: 0
Lli: 8214 Loss_of_sig: 0
Proc_rqrd: 94752 Protocol_err: 0
Timed_out: 0 Invalid_word: 1271378
Rx_flushed: 0 Invalid_crc: 16314
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 13
Suspended: 0 Lr_out: 18
Parity_err: 0 Ols_in: 0
2_parity_err: 0 Ols_out: 0
CMI_bus_err: 0

portstatsshow 4
stat_wtx 2098626822 4-byte words transmitted
stat_wrx 3740935752 4-byte words received
stat_ftx 1774821512 Frames transmitted
stat_frx 1936056545 Frames received
stat_c2_frx 0 Class 2 frames received
stat_c3_frx 1935961195 Class 3 frames received
stat_lc_rx 61959 Link control frames received
stat_mc_rx 0 Multicast frames received
stat_mc_to 0 Multicast timeouts
stat_mc_tx 0 Multicast frames transmitted
tim_rdy_pri 6757 Time R_RDY high priority
tim_txcrd_z 51203819 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 3260 0 51200559 0
tim_txcrd_z_vc 4- 7: 0 0 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0
er_enc_in 77821 Encoding errors inside of frames
er_crc 27107 Frames with CRC errors
er_trunc 64 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 10861 Frames with bad end-of-frame
er_enc_out 1193557 Encoding error outside of frames
er_bad_os 1534277 Invalid ordered set
er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout
er_tx_c3_timeout 6452 Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable
er_other_discard 79 Other discards
er_type1_miss 0 frames with FTB type 1 miss
er_type2_miss 0 frames with FTB type 2 miss
er_type6_miss 0 frames with FTB type 6 miss
er_zone_miss 0 frames with hard zoning miss
er_lun_zone_miss 0 frames with LUN zoning miss
er_crc_good_eof 16314 Crc error with good eof
er_inv_arb 135 Invalid ARB
open 0 loop_open
transfer 0 loop_transfer
opened 0 FL_Port opened
starve_stop 0 tenancies stopped due to starvation
fl_tenancy 0 number of times FL has the tenancy
nl_tenancy 0 number of times NL has the tenancy
zero_tenancy 0 zero tenancy


and this :

> PORTERRSHOW
frames enc crc crc too too bad enc disc link loss loss frjt fbsy
tx rx in err g_eof shrt long eof out c3 fail sync sig
=========================================================================================================

4: 3.3g 3.1g 1.4m 512.5k 288.7k 1.1k 0 225.2k 33.7m 106.6k 1 3 2 0 0

[Bob_Vance]

How far apart are the sites?

bv

[RUET]

12km

[Bob_Vance]

Well, that's more than the standard 10km.

You probably need to get long-haul GBICs for each end of the ISL
and you also need the extended fabric license on each switch.


admin> licenseshow
First Ports on Demand license - additional 8 port upgrade license
*** Extended Fabric license

admin> portshow 15
...
portFlags: 0x10004907 PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
portType: 18.0
POD Port: Port is licensed
portState: 1 Online
portPhys: 6 In_Sync
portScn: 16 E_Port
...
*** Distance: auto (desired = 40 Km, actual = 13 Km)
*** portSpeed: N4Gbps



I notice that your port just reports:

> Distance: standard <= 10km

so you do not have a long-haul and you're not <= 10km.

It also shows 2Gbs not 1Gbs



bv

[RUET]

Bob
thanks for these comments
regarding extended fabric, yes I suspect that this can be a good thing to have. But is extended fabric only allowing me to have more buffer credits ? and regarding stats I have, can we make a link to a lack of BB credits ?
REgarding line speed, I don't know why people who made this implementation had setup the ports to 2Gbs, I'm sure this doen't help ?
When you say long-haut GBIC, do you mean long wave GBIC ? Our borocade is connected to the leased line through a MRV box and they told us to have a compatible SFP DGD-SX/A which is I think a short wave ..
I have a qyestion regarding flow control. Do I have to enable ISL R_RDY Mode ?
pat

[Bob_Vance]

long-haul == long-distance

I was assuming that were using direct fiber.

But, looks like you're over DWDM.

I would think that the MRV people/documentation could help you.


I assume that you've seen this:

http://lordegg.wordpress.com/2007/05/18/r_rdy-potty-go/

[RUET]

Yes we are in DWDM mode ..
I will have a look at the link ..