Recommendations for accessing P4500 volumes from a Physical Server

[Peter J West]

Hi,

We're running a 6-node P4500 SAN.  The vast majority of Servers which are accessing volumes on this SAN are virtualised using vmWare ESX Server 4.

We now have a requirement to join two Exchange 2010 Mailbox Servers to the SAN as well and i'm just curious as to what peoples views are on how this should be achieved.

Our storage nodes are on a dedicated VLAN which means we effectively need to join the two Servers to two entirely seperate networks.

My original intention was to take the 4 NIC's on the DL380 G7 and split them into two teams of two.  One team would then be specified for use by the iSCSI Initiator, whilst the other would be allocated an IP on the data network and used for data traffic to and from the Mailbox Servers.

Has anyone got any experience of linking physical servers up to a P4500 installation?  We're currently seeing some strange issues with replication between the two mailbox servers (they're configured as a DAG) and i'm wondering if my config is wrong - or perhaps it's just because I haven't yet configured both of the Servers in the same way.

Grateful for any thoughts.

Pete

[oikjn]

sounds like the exchange servers are physical servers and not virtual.

Doesn't really matter either way and SAN/IQ doesn't care who is connecting to the LUN as long as the initiators are configured correctly.

Sounds like you haven't done much with iSCSI with windows servers.  I would be sure to read up on the documents with the windows action pack that comes with SAN/IQ.  You will definitely want to configure the HP DSM provider and maybe the VSS provider as well (I"m not runing EX2010 so I can't remember if snapshots are or aren't supporrted).

with 4 NICs and two physically seporate networks, I would setup two nics bonded as you like to your production network and then have the other two nics connected individually to the iSCSI network and let the DSM to the load balancing and failover.  I've done similar for hyper-v servers, physical servers, and virtual servers connected to directly to the SAN...  what you are going to have to read up on is how to configure iSCSI in windows and also how to setup the HP DSM and MPIO correctly.  Its not really a big deal, just follow the HP manual.

[Steve Burkett]

Yup, so in effect in WIndows you'll have three NIC's:

Production LAN (with a production LAN ip address)

iSCSI-1 (with a iSCSI network IP address)

iSCSI-2 (with another iSCSI network IP address)

You can remove all the Client for Microsoft Networks, QoS Packet Scheduler, File and Printer Sharing for Microsoft Networks etc etc components from your iSCSI NICs in Windows, and just leave it with Internet Protocol Version 4 (TCP/IP) enabled. No need to set a default gateway or DNS on these ones either.

Then use the Microsoft iSCSI software initiator to Connect to the P4500 LUN, once for each iSCSI NIC to create the two paths to the P4500.

[Peter J West]

Many thanks for the feedback.

@Steve You mention there's no need to specify the default gateway on the iSCSI interfaces - but is it a problem if we do so?  As an additional safety measure I was going to monitor ping responses from these interfaces and i've found if the default gateway ins't provided then we get no reply from our ping checks.

When connecting to the target is it the case that I need to use the 'Advanced' button and then choose the correct initiator IP?  When I do this at the moment I get two IPv6 addresses listed and then 4 IPv4 addresses.  Of course I could choose one of the IP's for the iSCSI interface but how will the system then handle the MPIO connection?  Is the driver intelligent enough to look for an interface on the same IP subnet as the storage nodes?

Cheers

Pete

[Peter J West]

Oh, and one more quick question.

Is there any way to view which interfaces are being utilised for the SAN connections?  When I run the iSCSI initiator admin tool I can see 4 connections but each of them goes to a different LUN on the SAN - this would suggest MPIO isn't being used and at the moment I have a single point of failure.

It would be handy to know which interface is being used when troubleshooting.

Pete

[5y53ng]

Peter, this is off-topic and I apologize, but I'm crious why you did not virtualize your exchange servers?

[oikjn]

there is really no need for your iscsi NICs to have gateway addresses and in fact you really shouldn't set them up with gateways unless you are going to have a remote VSA cluster also connected to the server.

What is your logic for trying to ping your iSCSI NICs?  are you worried about those NICs or the cables or the ports attached to those NICs suddenly failing?  If so, that is why you use MPIO with seporate cards on seporate switches.  If you are only worried about the server crashing, then just ping the production Team.

You really don't want to have multiple different gateways since you will likely end up having some of your outbound traffic running out of your iSCSI NICs instead of your Production ones.

You can easily see your network utilization using the network tab of the resource monitor.  Right there you will see the bandwidth usage per NIC.

If you setup the HP DSM MPIO and the iSCSI initiator correctly, it might appear at first glance in windows that you are only connected to on one NIC to one VSA.  The easiest way to check if you have it setup correctly is to look at the HP CMC iSCSI connection tab.  In there you should see two connections to one VSA IP (one from each NIC) and then you should see each NIC also form a connection to each VSA node IP that will say the Name/IP followed by DSM.  You should also see the same number of connections in the properties of the iSCSI initiator on windows, but the identifier ID will not be easily identifiable.

If you haven't set it up correctly... it can be confusing, so its often easier to disconnect everything, remove the favorite targets and start over.  Make sure the HP DSM is installed on the server and open the MS MPIO config page to make sure iSCSI is enabled.  Then when you connect to the LUN, make sure you check the box to use multipathing and favorites and then make sure to select "Microsoft iSCSI Initiator", select the IP of the NIC you want to use to connect, and then select the virtual IP of the VSA cluster.  Repeat the process a 2nd time exactly the same, but change the initiator IP to the 2nd NIC.

The reason you are seeing those extra IPv6 addresses is because you have some NICs with IPv6 enabled... find them and dissable them and those addresses should go away.

It can take up to a minute to have the extra DSM connections established.  If they don't show up after a minute in the HP CMC, you have something wrong.  Best thing I can suggest is get the manual for the DSM out and follow it exactly.  They do a clear walkthough and it works... follow it.

[Peter J West]

Thanks oikjn for your feedback - you raise some valid points and it's quite true to say we don't really have any need to monitor the interfaces.

I've gone through the documentation and along with your comments it's proving very helpful and I feel i'm moving in the right direction.

At the moment the CMC is showing 3 iSCSI sessions for this volume and I can clearly see from the Initiator IP/Port that two of these are from one of the iSCSI NIC's on the mailbox Server, whilst the other is from the other interface.  So at least I can now confirm connectivity from both NIC's.

The only problem I can see is that all three connections are to the same storage node and it's stayed this way despite me giving it a good 5-10 minutes to settle down.  Strangely the volume seems be to fully accessible so i'm not quite sure what's going on here.  Also none of the connections have the 'DSM' label against them.

You mention MS MPIO and i've confirmed that Multipath I/O is installed as a feature.  I've also confirmed the P4000 DSM for MPIO is present.  Reading through the deployment guide for P4000 DSM for MPIO I don't think there should be anything else required to get it working.  Is that not the case?

In the meantime i'm going to delete the configuration and restart it from scratch, just in case some of the previous config is being retained and causing issues.  

Thanks for your help so far - it's appreciated.

@5y53ng: I'm doing my config as recommended by our Exchange guys at head office.  They've suggested that virtualising the CAS role is fine, but the processor load on a mailbox Server is sufficient to warrant us running it physical instead of virtual.

[Peter J West]

I've found an application error logged relating to dsmctl2.exe. Sounds like it could be related and perhaps MPIO isn't being used because of this?

[Peter J West]

I've managed to fix it. :)

I spotted that the 'HP P4000 DSM for MPIO Control' Service was starting and then immediately stopping and this was creating errors in the log file relating to both DSMctl.exe and DSMctl2.exe.

A bit of digging around online led me to this article: http://support.microsoft.com/kb/2001997

I'd used the network adapter control panel in Windows to set the configuration.  The fix in the end was to simply open up the HP Network Utility and view the properties of the two SAN NIC's.  Both of them needed the tick in the 'Enable iSCSI Device' box.  Once I did this I also used the 'iSCSI IP Settings' button to confirm the IP address of each interface.

As soon as I did that the Service started without problems and i've now got at least 2 connections to each node in the SAN that is located on the same site as the server.  I notice one node has two additional connections but I seem to recall that being normal as one node acts as a 'gateway' for when MPIO isn't being used?

Thanks for the comments - really pleased to have got a better understanding of how this stuff works.

Pete