BL680c G5 w2k3 64-bit - Bad NIC's or User Error?

[Erik Briggs]

Ok, so this last weekend I tried to go live on my new blade server and was forced to roll-back to my old server because I couldn't get this BL680c working correctly.

I'll explain first what I was trying to do. We needed a W2k3 R2 64-bit Server running in a cluster with SQL 2005 on it. We need at least 2 NIC's, 1 for LAN and 1 for iSCSI.

Now, I'll run down what I did, and what I found:
I configured our 2 GbE2c switches for our various VLAN's and got everything talking as it should. I got windows installed easily, but first had problems adding it to our domain. I was getting all sorts of DNS-related issues. I could get the computer to add to the domain, but not change names. I could not get the cluster service up and running. After trying to change names, and removing it from the domain, I could not get it to re-add. While it was on the domain, I would get 5 minutes of "Applying computer settings" and then when I would go to log onto the domain, I would get another 5 minutes of "Applying user settings." On an upgrade weekend, when nothing is working, having to wait 15 minutes per reboot was excruciating. After no sleep and working 20 straight hours on it, I decided it wasn't ready, so I rolled back to the old server.

I decided then, since the pressure was off, to reformat and start again clean so I could attempt again this upcoming weekend. I ran into the same issues, but was able to dig deeper. I tried all sorts of things to fix my DNS issues (since anyone who has seen those problems knows that it means that the PC can't talk correctly to the DC's and/or DNS).

After absolutely nothing worked, I decided to check my network configuration. I had configured the 2 Dual Port NC326i NIC's in a Team to connect to the LAN side. The first thing I did was remove the team and disable one of the NIC's. The other NIC's weren't teamed, so I just disabled one of them for consistency. When that still made no difference, I decided last night, as a last ditch effort, to setup one NC373i as my LAN adapter, and one NC326i as my SAN adapter (with the other one of each disabled (no teaming)). Immediately, my domain/DNS issues disappeared. It was a miraculous change of events. I then decided to setup iScsi on the NC326i. I was able to mount my targets, but as soon as I tried to do anything with them, the Event log got chuck-full of iScsiPrt errors. The iScsi connection was being dropped and reconnected very quickly. This helped me to understand what was happening to my lan connection when this same adapter was configured for the LAN.

So, to fix it all, I disabled both NC326i PCIe Dual Port Gigabit Server Adapters, and configured one of my NC373i's for LAN and the other as SAN. I don't get any redundancy this way, but at least I am working and will be able to successfully cut over to this server this weekend. After doing this, all my cluster and SQL config went flawlessly.

So, the question is: Is this the fault of those adapters, their drivers or firmware, or is the problem mine for trying to use them that way. I am not sure I understand what purpose they are supposed to serve, but its possible I did something wrong. Anyone who can help point me to the light please let me know.

BTW, the drivers and firmware are completely up-to-date on it 10.85.0.0 drivers and 2.1.2.0 firmware.

[Adrian Clint]

The 680 has 4 onboard NICS. 1 & 2 are Multifunction (accelerated iSCSI) 3 and 4 are normal NICs.
Two (2) embedded NC373i Multifunction Gigabit Server Adapters with TCP/IP offload engine for Microsoft Windows iSCSI boot, and optional accelerated iSCSI
One (1) NC326i Dual Port Gigabit Server Adapter

Don't let the "i" in the name let you think its got something to do iSCSI - I think its for "i"nternal.

I would think that its best that you make your NC373i NICs the iSCSI SAN connections. At least you can then accelerate the performance with the iSCSI license if you need to. You cant do that if you are using the NC326 NICs for SAN.

Start off with LAN on a single NC326 and SAN on a single NC373 and see how that works. Then try teaming them.