Upgrade advice - Take a quick look and please give me yours!

[GaryF]

Hello to anyone looking for a Friday afternoon diversion (or whenever your time zone puts you).... I have a small network where continuous uptime is *critical*. It's worked well for several years, but is in bad need of an overhaul.

Currently, I'm running my entire LAN (one segment) over 4 venerable ProCurve 4000m switches in a fully meshed topology, each connected with 1GB-SX "mesh" trunks. Two are in the data center, where I've got teamed NICs in my main servers connected one to each switch. If one fails, the servers are still connected through the survivor. The other two switches are in the closet serving the desktops. If one fails there, I move the wires off the dead switch into the spare capacity on the survivor, and am back at full strength quickly. I've never had a switch fail in 7+ years, but the resiliency has given me (and management) a lot of comfort. As a layer 2 network on one segment, it's really easy to manage.

Now, I need to upgrade for (1) more bandwidth, especially in the data center; (2) better security, by moving some functions out of the LAN into screened subnet/DMZs. We'll be deploying VoIP on the LAN soon, so high availability will be even more important, if that's possible.

My proposed solution to this is to (1) replace the 4000m with something faster, and (2) get a solid routing firewall box (ideally redundant) to manage the routing and access to the various DMZs and the Internet. (At the moment, I've got the Juniper SSG 140 in mind, but that's another topic.)

So, putting aside the firewall issue for the moment, I see these options to upgrade the LAN (1):

(a) Drop in faster switches with mesh capability. The pros are that this maintains the robust availability and simple administration. The cons are that HP has limited this function to the 5400zl and 5300xl series, which are on the high-end of what we can afford. Plus, meshing knocks out all the (potentially) valuable routing features of the box. If I'm using the firewall to route between my few VLANs, though, I'm not sure this is a big deal. Just seems like a waste to me. (HP - how about including mesh functions in the 4200vl ?!?) It's also interesting to me that, unless I've missed it, HP's competition doesn't offer anything this simple and effective for the small environment. You've got Foundry's VSRP, for example, but it's necessarily more complex....
(b) Create a more generic topology with 2 switches in the data center, and 2 in the closet. Run trunked 1Gb links (10GbE is too expensive or N/A) between the four in a ring topology for redundancy, and use Spanning Tree to manage the redundant links. So, this may be cheaper out of pocket to start up (4 x 4200vl-48G vs. 4 x 5400zl-48G ~ $10K), but it's tougher to configure, and seems like fail-over performance is not necessarily fast or guaranteed. Also, I appears that teamed NIC members can connect to different physical switches in this environment, but can anyone confirm that to me?
(c) Do like (b), but use layer 3 facilities like OSPF to manage the routes. Of course, this doesn't save money (in the HP line, anyway), and it certainly complicates things.

So, if I understand these options correctly - and not being too experienced with (b) or (c) - it sounds like (a) is the way to go, even if it comes at a relatively steep price to start. Am I missing something simple; is there a flaw back in my basic topology (LAN/firewall/DMZ); or does this sound reasonable?

I'd really appreciate your feedback. Thank you!!
Gary

[Matt Hobbs · Accepted solution]

Hi Gary,

I totally understand the difficult decision making process you're going through right now.

If price was not an issue I would go for the 5400's throughout the network. I would get 2x5406's to act as your redundant layer-3 gateways with the Premium Edge License which gives you VRRP, OSPF and PIM. Depending on how critical Internet access is for you, you could just get away with the one firewall.

For the other 1-2 switches I would also use 5400's. (You have a lot higher port density so you could actually get away with only 1.6x 5400's vs 4x 4000M's).

I would use spanning-tree for redundancy simply due to the fact that routing and meshing cannot be enabled at once. (However, if concurrent meshing and routing ever happens it will only be on the 5400 series).

For your NIC teamed servers, spanning-tree is fine.

In regards to meshing, the current incarnation on the 5300 and 5400's is slightly different to that of the 4000M's. For end-nodes, it works pretty much the same (from memory it drops the first packet from an unknown source-address so that the entire mesh can learn it). For actual switch management though, in a larger mesh when using telnet/ssh to a switch a few hops away, it will often time out while going through this learning process. On the second attempt it will connect. This is because the learning process for mac-addresses of the switches themselves is slightly different. Once the connection has been established though, no further packets are dropped.

This little gotcha really bugs me which is a shame because I love the concept and simplicity of meshing. I have been told that the reason for this change in meshing was to further increase the stability and resiliency of the meshing protocol.

So depending on how often you are actively managing your switches and starting up new telnet/ssh sessions, this caveat may or may not bother you.

Other solutions you could go for as you've suggested would be:

2x 5300's with XRRP for your layer-3 and some 4200's at the edge using spanning-tree. In this routed design, I would use 5300's instead of the 4200's if possible.

or

4x 5300's with Meshing only.

In either of these 5300/4200 solutions, bear in mind that the 16-20 port Gigabit modules are oversubscribed to the backplane. So at minimum, for your switch to switch links I would recommend using only the 4-port Gigabit modules.

With the 5300's you also get that benefit of XRRP, OSPF and PIM without the need for a Premium license.

It really is a case of trying to balance out what you need now and what you may need tomorrow. The worst part is when you know you could have bought something that had a feature you thought you'd never need, but one day that need does come up and boy do kick yourself then!

The 5300's and 4200's are really at the end of their development life now, so it's unlikely we'll see many new features on them. Whereas the 5400's will still have a few more tricks up their sleeves.

One other thing I like about the 5400's is the removable fan tray and management module. It's simple things like that make your life a lot easier if you do ever see a failure.

Matt

[GaryF]

Thanks for your reply, Matt! I really appreciate your insight on the product line and its use. Like you, I love the "concept and simplicity" of meshing; and prior to your response, my gut feeling was to just use the 5400's as fast layer 2 switches in a mesh configuration. I'm now leaning more that way, and think these are my main concerns driving that:

(1) Meshing has essentially zero-latency fail-over, since all the mesh links are active; and I have the impression that STP does not (though it seems all the vendors have taken steps to make it "fast"). Unfortunately, we have some legacy applications that aren't very tolerant of even brief server disconnects or delayed response, and those are line-of-business apps that are in use in real-time. You said "For your NIC teamed servers, spanning-tree is fine", but I'm not clear from my reading as to how well it would perform in my situation. Maybe you can point me to a good resource or correct my understanding, if I'm off-base.

(2) I can tolerate some network disruptions in the routed subnets (Internet DMZs, etc.), though people will complain fast and loud. But downtime in the LAN is truly unacceptable. The simpler it can stay - then the better it should perform (and the fewer the opportunities for me to screw something up).

The vanilla 5400 and 5300 are about the same price at 48 gig ports, and the 5400 looks clearly like the better value if you don't need the extra-cost options. You've confirmed that it is the preferred product, so I'm good with that. Guess then I'm paying 2X the price of a "normal" layer 2 switch just to get the mesh. On the other hand, the 5400 performance specs are clearly high-end; it does give me PoE (which will only save me a few hundred dollars, but it's something); and it will have more future upside with HP.

I was approved for new switches to handle the DMZ connectivity, but I could probably use the old 4000's there instead, and put that money toward the four 5400's. And, as you said: "The worst part is when you know you could have bought something that had a feature you thought you'd never need...." Hey, maybe they'll eventually blend the meshing and layer 3 feature set (and when 10GbE gets affordable, I can use the nice 10Gb-certified fiber riser we have, too).

Thanks, too, for pointing out the "simple things" like the fan tray. The only fault I had on a 4000 was a fan, and we lived with that for a couple years rather than take part of the LAN down. (When we relocated the office, I replaced the fan packs in all the switches.) Indeed, a hot-swap fan was high on my list of hoped-for features!

So Matt, thanks for your time on this, and I'm all ears if you have any other hints. I admit I don't truly see the value of the layer 3 functions in the LAN switches for me yet, so illuminate me if you will. Even so, I'd lean toward simplicity today, with the hope of utilizing the bells and whistles tomorrow. You have definitely given me some comfort with the direction I'm headed, and your contribution has helped me focus the objectives.

Thank you!
Gary

[Paulen]

Gary,

Depending on the size of your network, I really would try to avoid using meshing - it really became a pain in our network, and it is "impossible" to troubleshoot. I'd rather use routing. Its much easier to maintain.

[GaryF]

Hi Paulen,
Size-wise, I'm looking at 4 x 48 port switches, and anticpate adding 2 more x 48 port max in time (2 in the data center, 2 for each of two floors above).

On the meshing, I'm just going off my experience with the 4000m switches that I've been running for 7+ years, and they've been truly trouble-free (4 switches, fully meshed with both 10/100 and later with Gb-SX fiber).

Point taken on the troubleshooting - I fortunately haven't had to do any... What models are you using, and how many, etc.? Do you think this is universal trouble?

Thanks for your input!
Gary

[Paulen]

Hi,

I used 5304xl, 4 and 5 in each mesh. We are trying to move them towards the perimeter of the network, due to generally buggy behaviour. We are migrating to 5406. We have ~100 5304xl, 200 HP2626 (which really is a bad switch, only good thing is price) and 150 25xx. The most stable switches we have had are the 4000 and 25xx. The 53 (and 54) have had serious bugs, and the 2626 just stops..

[GaryF]

Well, you've got a *serious* switch investment there, Paulen! I've never had to deal with more than a half-dozen devices - fortunately, I suppose. My ProCurve experience has been excellent, particularly in contrast to some buggy gear that I inherited before it. I don't want to return to that!

HP ProCurve has had my trust as a preferred vendor. The 4000m's earned that trust with their simple to admin, reliable meshing feature for H/A; lifetime software/hardware warranties; and supremely stable operation.

Your feedback is unsettling...(and my experience with the ProCurve design group, while previously quite helpful on another issue, has been disappointing on this one). If the new, more complex devices aren't as rock-solid, and if meshing will cause more problems than it solves, then I've got to broaden my view (e.g. cis.../foun...).

After all, 4 x 53's meshed sounds about as simple as can be. Perhaps they've bitten off a bit too much with the intelligent edge? Unfortunately, I won't really know until they're bought and running.

Thank you for your time and input!
Gary

[Paulen]

Gary,

Don't get me wrong - compared to the size of our network, and the money spent on gear, HP ProCurve is outstanding. Its just that we might have been stretching the limits somewhat, thus "pushing it".

We use the switches in our distro network, have Juniper MPLS core and Allied DSLAMs. We serve ~15.000 private customers, some big hospitals, municipals etc, and it has been an overall success. BUT when something fails (and it *will* fail..) it is a pain in the *ss to troubleshoot.

I would go for the 3500/5406. It seems like a solid thing, despite some issues. But then again - show me a C*isco switch without bugs.. You have unparallelled price/performance, and can do pretty neat stuff with it. But if you really want to do some routing, I'd use something else. Not that it won't work - its just that I am not comfortable with the toolset available via cli to monitor, troubleshoot and supervise.

That being said, we run OSPF on 3424cl flawlessly. There is no such thing as a straight answer :o)

Good luck in your process. I will be more than happy to share my experience with you.

/Bjorn Tore

There

[GaryF]

Many thanks for the follow-up, Bjorn. Yeah, I don't want to be too wobbly on this... I'm coming from a small company environment, obviously, with about 80 people involved. We serve ~ 4000 customers with inbound call center services, and if the network stops, the whole business stops dead - and customers are immediately affected. You're in the same boat on a much larger scale! However, these boxes are my core and edge all in one, so a fault kills everything.

With this upgrade, I'm messing with something that has worked like the water faucet for the past 7+ years. Just plug it in - and there's the network. The meshing has in theory given me great resiliency, but it's not truly tested. Who knows - maybe if a switch died last year it would've crashed everything. Fortunately, I haven't had to deal with that. My admin of the "network" was to periodically check the switch event logs, and look at the mesh port counters to see that they were reasonably even up.

So now, I'm trying to balance the need for more performance and scalability with my ability (time, really) to fully understand and monitor the additions. Plus, dropping voice on the LAN makes failure that much more painful. My past experience was so good with the 4000 mesh, that I'd hope to drop in the 5400 in the same way, knowing that it gives me speed and capabilities for later. I'd deal with a capable firewall for the (relatively simple) routing and access control. When I start hearing that the meshing may not be that great on the new gear, then I'm looking at a learning curve for STP and routing. I think those are and will be useful things, but I'd like to defer that until later.

Right now, I'd be happy with simply getting a speed upgrade by replacing the 4000's, if it otherwise works as well as what I have now. If there's a hardware/port/cabling problem, I feel that I've got few enough devices that I can deal with it. If the 5400 will have trouble running a "simple" 4 to 6 box mesh reliably because of software issues, though, then that's very unattractive. I never rebooted my 4000 switches other than for a rare firmware update (maybe twice in 7 years). My core servers typically run for 2 to 3 years before I schedule any downtime, and may go longer. Switches that need to be rebooted on any regular basis will be very unpopular - especially since I'm getting them to spend 3 - 4x what we spent on the 4000's many years ago... So, I suppose if you can give me any insight on particularly what problems you've seen in the 5400 that you think are related to meshing, then that would be helpful.

It's probably still my preferred route, but I really do need it to fire up and just work. Fancier stuff later.
Thanks again!!!
Gary

[Matt Hobbs]

I will vouch for the stability on meshing on the 5300's. I have worked with one customer running 4x 5308's in a full mesh, fully populated with the 4 port gigabit modules each going to 2600's at the edge. From each port on the 5300's there is approxiamately 200Mbit of traffic 24x7 and it is rock solid.

I shouldn't think things would be much different with the 5400. Especially in your setup where things are quite basic I don't think you'd run into any serious problems.