A few basic SAN questions

[Mikko Peltoniemi]

I'm new yet to SANs, but I'm looking into upgrading our storage into a SAN solution in the near future. Sorry if this boring basic stuff, but I have to start somewhere... :)

A little bit of background first - I have three buildings, connected together via fiber optic cable. The Ethernet on the fiber is 1 GB/s. I'm using ProCurve and 3com switches. Although I'm planning on replacing the 3coms with Procurves as well.

Now - I'm interested in a SAN via IP solution. What seems like a great plus is that I could fairly easily put another array in another building, and use it for redundancy. Primarily storage will be in the same room as the server though.

But since I have an existing Ethernet already connecting the buildings... Plus, I do have the option of upgrading the GBICs to 10 GB/s models.

Or would I still be better off going with fiber channel instead? Either forgoing the redundancy using other building, or by perhaps using an extra pair in the cable for the SAN.

But in my mind the IP solution is more cost effective, since the basic transport infrastructure already exists.

I don't have any heavy bandwidth needs. File sharing, MSSQL, email... About 40 users, but they're split between three buildings so wherever the storage is, about half the users are accessing it over the fiber.

Any comments? Suggestions?

Thanks!

[Zygmunt Krawczyk]

If you are cost concerned, good solution will be iSCSI SAN

1) HP P2000 G3 iSCSI dual controller
http://h18000.www1.hp.com/products/quickspecs/13551_div/13551_div.HTML

or

2) HP P4000 G2 SAN Solutions
http://h18000.www1.hp.com/products/quickspecs/13552_div/13552_div.HTML

>> But since I have an existing Ethernet already connecting the buildings... Plus, I do have the option of upgrading the GBICs to 10 GB/s models

Which type of fibres are building connected (multi-mode or single-mode)?

[Mikko Peltoniemi]

I don't know yet how much of an issue cost is. That's what I'm trying to figure out though.

We have a multimode fiber connecting the buildings, right now I'm using the ProCurve SX-LC Mini-GBIC (J4858C) for connectivity, but I think I could upgrade that by using the 2-port 10 Gigabit AL expansion module plus the appropriate multimode transceivers...

I have extra pairs, but I might need them for future phone expansion, so I would rather avoid using them...

[Zygmunt Krawczyk]

10GbE on multi-mode fiber has limited cable distance. In the best case you can reach 300m max using high grade multi mode fibre (MMF). This distance you can reach using 10GbE SR module with 50/125 microns / 2000 Mhz*km MMF, or using 10GbE LX4 module with 50/125 microns / 500 MHz*km MMF. See the table 1 in doc

http://www.cisco.com/en/US/prod/collateral/modules/ps5455/ps6574/product_data_sheet0900aecd801f92aa.html

[Mikko Peltoniemi]

I am well within the 300m limit... Fortunately the buildings are right next to each other and the cable goes almost straight through.

But of course there could be other unforeseen problems.

Good point though. I certainly don't want to spend all that money and then find out it doesn't work.

I wonder how I could be sure of that without actually buying all the equipment first...

[Zygmunt Krawczyk · Accepted solution]

You need to know, what is the core size (50.0 or 62.5 microns) and modal bandwitdh (160, 200, 400, 500 or 2000 MHz*km) of the fibre cable. If you do not have the documentation, you can order the specialist with proper equipment to measure these parameters.

[Don Mallory]

Running FC over IP is not really an option, and given the work on Converged interfaces, you would be better off with either an array that is capable of both (including FCOE), or just going with iSCSI.

iSCSI works quite well, however the primary gains against FC will require you to use 10Gb.

Next challenge. iSCSI, like FC and any block level SAN infrastructure must be segregated, and must be redundant. That means, different switches, ethernet cards, IP ranges, etc. to both maintain stability of the environment, but also security of the data.

What iSCSI does is bring all the benefits of SANs to the cost effectiveness of IP/Ethernet, while also bringing all the security risks of Ethernet to the SAN. You also cannot realistically tunnel using IPSec or other VPN technology without severely impacting performance.

The HP P2000 series (Lefthand arrays) are a good offering, Dell Equalogic (they bought that), also don't forget converged / unified storage arrays such as the EMC Celerra, NetApp, or the HP EVA arrays.

[Uwe Zessin]

MSA2000/P2000 = DotHill
P4000 = Lefthand Networks

[Mikko Peltoniemi]

Thanks all, it's been very helpful!

[Thomas Callahan]

[quote]
Running FC over IP is not really an option, and given the work on Converged interfaces, you would be better off with either an array that is capable of both (including FCOE), or just going with iSCSI.
[/quote]

Why exactly are you saying that FC over IP is not an option?

[Don Mallory]

Hi Thomas,

FCIP or iFCP, depending on whether you want a split fabric or a single integrated fabric is really only used to extend large fabrics. If you have everything in a single data centre, great, no need, if it's over 100km away, you need these to interconnect the fabrics, but to do so will also require either a Multi-Protocol Router (MPR) or a director with modules that can do that sort of thing.

The cost of an MPR (or blade), and/or licenses to use those services, plus the SFPs for interconnect and the WAN bandwidth to support it, you will find that it's not cost effective.

By the time you get to this level, it's better to look at FCOE (Fibre Channel Over Ethernet), which is not the same thing at all. At the point that you get into FCOE, yes, the cabling is less expensive, and converged switches are less than fibre switches, but once again, the cost of FCOE HBAs is notably higher than a couple of extra NICs with or without TCP Offload Engines (TOE) to support iSCSI and also noting that a lot of (I)OSes don't seem to support TOE very well yet, at least not without tweaking.

Yes, I agree that some of this is a sweeping generalisation, but given that Mikko is new to implementing SAN infrastructures and is in a SMB style office setting (40 users), getting into a FCOE implementation seems a little overkill when iSCSI would more than likely meet the needs as they have been identified.

Don