ALB stp switch configuration

[danletkeman]

Hello,

I am looking at changing our p4500 nodes from LACP to ALB.  We have dual ISCSI switches for these nodes, and I was wondering what everyone does for management redundancy.

Example:

node1 - nic1 connected to switch 1

node1 - nic2 connected to switch 2

node2 - nic1 connected to switch 1

node2 - nic2 connected to switch 2

Best practice for ALB is to disable stp and leave the switches with an air gap, or not interconnected.  Ok, so you still need to have a route to mange the nodes with the cmc.  If that route comes from switch 1 and switch 1 happens to go down, you will loose management to the san.  As a secondary option I could manually add the route to the second switch if need be but that seems like something you shouldn't have to do.

I was thinking about leaving stp on and configuring and hsrp vlan between the switches and routing the traffic out both.  Would this hinder or cause any path problems keeping stp running?

[Emilo]

I am not sure why you would want over complicate this?

it is a very simple setup.

Flat network.

No Spanning tree leave that for your swithes and routers.

Two swithes with a a trunk between them.

Setup your NIC's in a ALB bond for failover and throughput

Dont get cute.

[David_Tocker]

I think you have two questions here:

1) STP.

2) HSRP.

1) STP is really only an issue in the delay of 'live-ing' up a port (or not live-ing it up at all).

2) HSRP will cover your routing redundancy.

I have not personally concerned myself with routing redundancy. I have only ever seen my switches die completely, not just the routing side of it. As we use stacked switches (cisco 3750g/x/e) it somewhat overcomes this issue as if the routing 'master' fails it will fall over to another switch in the stack that will take over the routing role.

STP is recommended to be disabled on the -ports- that your nodes are connected to (and any trunking ports/vlans in between them) in order to promote a fast failover and prevent issues with blocked ports etc.

If you are running good (read:expensive) stacked switches then you should be able to do this on a per-vlan/port basis.

If you have cheap switches then I suggest that you dedicate them to iSCSI alone and break out the vlans to a seperate redundant router/switch stack to perform the inter-vlan routing. Actually in hindsight it seems like a bloody good idea.

Sometimes standards based systems such as HSRP can be a bit unwieldy, I would seriously consider the stacked 3750 solution, I sleep well at night....

[danletkeman]

David,

Thank you for the excellent reply.

I am using 3560G's dedicated to ISCSI (yes in hindsight it would be a good idea, I wasn't before and now I am).  I don't have any 3750G's to stack, but we are planning on replacing the 3560G's with a 4500E chassis in the next 10 months.

I think I will try disabling stp on the iscsi vlan and leave hsrp out of the mix.  Probably won't have any problems with the switches anyway.

[danletkeman]

Not sure if I understand the need to trunk the two switches together, each node is connected to each switch and each esx host is connected to each switch, so why to we have to trunk the switches?

If I trunk the swtiches then the iscsi traffic goes like this:

esx host ---- switch ----trunk ---- switch -----san node

when I would like it to go like this instead to it doesn't traverse the trunk:

esx host ----- switch ----- san node

This seems to be very troublesome on a non stacked switch pair, because I'm seeing drops on the etherchannel between the two switches.  

But if I remove the trunk I loose quarum on the san and everything is dead.  At this point I am thinking of ditching the dual switch setup in favor of less output drops and faster performance by leaving everything on one switch.

I must be missing something here....

[Emilo]

You need a trunk because each swtich needs to know about the other mac address table.

That is why you need either a "stacked" swich or a trunk in between them.

Trunk links are required to pass VLAN information between switches, A trunk port is by default a member of all the VLANs that exist on the switch and carry traffic for all those VLANs between the switches. To distinguish between the traffic flows, a trunk port must mark the frames with special tags as they pass between the switches

[danletkeman]

I do understand the reason for the trunk and how trunk ports work.  My question is more related to how he san works.

If node 1 has a connection to each switch and node 2 has a connection to each switch it should never have to traverse the trunk port for iscsi traffic.  Each san node can see each san node and each esx host can see each san node.

I did some more reading and found out that each lun is only served by one gateway node, so this may be the reason why it has to pass through the trunk port, but even with this as a requirement the vmware host should still be able to get to the gateway node via either switch....

I understand if you would have dual switches and have the nodes in LACP mode that the traffic would have to traverse the switch trunk because each node is only connected to one switch, but wtih ALB it just makes no sense why the traffic would have to go accross the trunk.

Stilling thinking that I will have to ditch the dual switch setup in favor of no trunk and less output drops.  Will be moving to a chassis based switch shortly which will solve all of these issues, I just wanted to find out if there was a way to not have the traffic pass through the trunk ports.

[Jay Cardin]

You may want to go back and recheck the best practicies for ALB.  http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c03035314/c03035314.pdf

Look on page 64  "Figure 28 Adaptive Load Balancing in a two-switch topology"

There is no "air-gap" for ALB.  The two iSCSI switches should be connected together.

[David_Tocker]

danletkeman,

The nodes communicate with each other directly. This is why there cannot be an 'air-gap'

If you are having drops on your trunk then you should consider aspects such as:

1) is the trunk big enough (multiple ports)

2) do you have flow control enabled on the trunk? (you should AFAIK)

when you say 'drops' what are you refering to? RX/TX errors?

[danletkeman]

4Gbps etherchannel trunk.

I enabled flow control on the trunk now, guess we will see what happens.

They are always TX(output drops).

Thanks,

Dan.