Brocade ISL

[John Carver]

I have 4 Brocade Silkworm 2800 switches setup in a cascade topology. There is a single ISL link between each of them. One link in particular is showing excessive usage:

0x1023fc60 (tThad): Apr 3 22:11:44
WARNING FW-ABOVE, 3, eportTXPerf004 (E Port TX Performance 4) is above high boundary. current value : 95462 KB/s. (faulty)

And then a normal message is logged:

0x1023fc60 (tThad): Apr 3 22:11:52
WARNING FW-BELOW, 3, eportTXPerf004 (E Port TX Performance 4) is below low boundary. current value : 12591 KB/s. (normal)

This condition bounces back and forth during our peak production times. Is this just a bandwidth overload of the ISL? Will a second ISL connection reduce the load on the first just by plugging in the cable and letting the switches auto configure the ports to E ports?

[Peter Mattei]

Yes, it looks like the ISL is just reaching its limit of 1Gb = 100000kB/s.
And yes, if you connect a second ISL it will automatically be configured as E-port unless it was set to a fixed value. But you will see it.
The load will be statistically distributed between the ISLs at FC login time.

Cheers
Peter

[rich pattison]

If you're getting performance issues on your SAN, it might be a good time to review the topology and implement a more efficient configuration such as a meshed environment.

It could also be worthwhile identifying traffic patterns to try and localize high data rates to the same switch, to reduce ISL usage.

[Peter Mattei]

Well, if you would start reviewing your SAN I would consider replacing your rather old switches by new 4Gb ones.
These will be faster and maintenace cost will also be substantially lower.

The 4Gb switches are available from 8 to 64 port versions. Each one is available with flexports.
Examples:
The 8 port 4/8 switch has 16 ports; 8 enabled with 4 port upgrade options. At the end you will have a 4/16.

The 4/32 switch is available with 32 or 16 enabled ports. You can upgrade in 8 port steps.

Cheers
Peter

[Mark Poeschl_2]

Note that to "automatically" form a trunk (load sharing) ISL both ports in the ISL have to be on the same ASIC or "quad". Quads come in groups of four - ports 0-3, ports 4-7, and so on.

With those older switches I doubt you'll run into licensing issues, but some models did not come with a Trunking license by default and you actually had to buy it separately.

[rich pattison · Accepted solution]

..........2800's don't do trunking......!

Rich

[John Carver]

Do the 2 ISL connections need to be in the same quad of ports since we are not configuring for trunking? Now the bandwidth saturation errors are showing up on the new (second) ISL.

[rich pattison]

My understanding is (off the top of my head) - the 2800's had dual port ASICS - not 'quads' - each pair were adjacent ports (0&1, 2&3, etc) - so it follows that you should connect the second ISL into the adjacent port. However I don't know if this is a "requirement"...........or if I've mis-remembered something.

If I can find out some more information - I'll post it here.

Regards
Rich

[Peter Mattei]

The 2800 does not have trunking capabilities so it does not matter where you put your ISL.
Trunking has been introduced with the first 2Gb switches.

>> Now the bandwidth saturation errors are showing up on the new (second) ISL.

Check the throughput of both ISLs. Are both hitting the ceiling of 100MB/s or just one?

Without trunking a single FC exchange will only use one ISL and you cannot tell which one it will take. This is negotiated at FC login time.

Are you doing backups over this ISL?
Backup is the dominant MB/s "producer". If possible kepp backup traffic local, i.e. on the same switch: Backup source host on the same switch as tape library!

Cheers
Peter

[John Carver]

Only the second ISL is approaching 100mb. Running the command portPerfShow reveals that most of the time it's running around 30-60mb with spikes close to 100mb while the first ISL runs around 1-2mb. If I disable the second ISL port the first one of course takes over but as soon as I enable the second ISL, it once again carries most of the traffic. Backups were not running during this time of observation.

[rich pattison]

You've just highlighted the difference between multiple ISL's and trunking. Multiple ISL's act independantly and do not load balance, whereas trunks are designed to aggregate the bandwidth of the individual links.

The one way you may be able to affect performance would be to use zoning to force individual HBA's to use a specific path (ISL), but this assumes the problem is with multiple servers all trying to use one link. If the problem is with traffic from one server, then zoning may not help.


Regards

Rich

[Uwe Zessin]

> use zoning to force individual HBA's to use a specific path (ISL),

Last time I checked, zoning was ignored by E_Ports and thus could not be used for traffic engineering.

Can you please describe the necessary configuration steps to do it?

[John Carver]

Through a series of tests I just finished I can verify that that "E" port zoning doesn't seem to work. I even removed the ISL ports from the zone and traffic seemed to know how to make it from one switch to the other.

[Uwe Zessin]

I have NEVER put E_Ports into zones during the last 7-8 years and always had connectivity across ISLs.

[Peter Mattei]

ISLs cannot be included in zones.
The only thing you can do is assigning ISLs to ports (not recommended) !
However with the behaviour you described I assume you have one stream with heavy IO.
Try to find out the source and destination port and then put both on the very saem switch (if distance allows)!

Cheers
Peter

[rich pattison]

You certainly can't zone E_ports. If the ISL's are between the same two switches, there's not a lot you can do (as seems to be the case here) - but in a cascade, it may be possible to split traffic down different paths using zoning, if HBA's and storage ports are spread across different switches.

I'm not sure if the topology described is a mesh, or a cascade (if each switch has one link to every other switch, its probably a mesh), but if a path between two switches is congested, then maybe moving busy ports off those switches, and onto others - then using zoning to align the HBA's & storage ports may be possible.

Rich

[John Carver]

Since it appears that there's not much I can do to balance the traffic across the 2 ISLs, I went ahead and reconfigured the heaviest traffic to remain within the same switch and not cross through the ISL. The upper limit warnings have ceased. I think faster switches that also allow for trunking are in our future. Thanks to everyone for the help.

[Patrique]

Good afternon, I think that another ISL will resolve your problem, but what you are trying to do is called ISL Trunking, it's limited do 4 link's (4Gbx4= 16Gb/s), but to do that you will need a license to make it work. I hope that was helpfull (sorry my bad english).

PT

[Peter Mattei]

Patrique

If you read the question carefully you will notice that we are talking about Brokade Silkworm 2800 switches that do not have the trunking capability :-(


John

Glad to hear that your could redistribute your Hosts.


Cheers
Peter