Services fail to start with HP NIC Teaming

[Jeffry A Spain]

I enabled HP NIC Teaming on a number of our servers. On systems where services log on with a domain account, rather than with the local system account, those services fail to start when the system is rebooted. Creating a service dependency on a late-starting service such as Print Spooler doesn't help. They can be started using services.msc after logging onto the system. Those same services do start normally after a reboot if NIC Teaming is disabled.

The systems are all running Windows Server 2003 SP-1. The HP NIC Teaming driver is cpqteam.sys, v8.10.00.0, which is part of the most recent HP ProLiant Support Pack 7.40b.

My hypothesis is that this is a timing issue. It probably takes longer for the HP NIC Teaming driver to establish network connectivity than it does for the drivers for the physical NICS when they are not teamed. Thus with NIC Teaming enabled, the services logging on with domain accounts are trying to start before network connectivity to the domain controller is available to authenticate them.

My question is how to block these services from starting until full network connectivity with HP NIC Teaming is established.

[Robert S. Carr · Accepted solution]

Jeff:

If you're having NIC Teaming problems, you are not alone. I too wonder why MS OSes have had a problem with teaming as it seems to work well in one place and fails in others. Teaming is often used in conjunction with clustering so with other uses it may not work that well. :(

To answer your question, to the best of my knowledge there is no way to slow down the start up of services. There is a way to start services at bootup. With a batch file you can use the command sleep.

For example,

sleep 10

makes a batch file wait 10 seconds before proceeding.

So you could have a batch file called SrvStart.bat in the Startup folder found in the profile for All Users. It might look like:

net stop spooler
sleep 20
net start spooler

This will shut down the spooler (which is failing anyway) then start it again in 20 seconds. Clean shutdown, clean startup.

This could be a temporary fix but the true problem lies with teaming itself. Here are some suggestion to try:

1) Use a static IP address for the server not DHCP. Do not rely on DHCP from another server or the booting server itself. Using DHCP from a server is slow. Using it from a switch or router the server it is connected to is much faster but a static address is fastest. Suggest you use that.

2) Use a static speed for the NIC. Do not use Auto Negotiate. Once again, it's a time waster. Set it to the correct config. e.g. 100 Mb/s Full Duplex

3) Teaming is very sensitive regarding DNS data. Two adapters must use the same DNS data. If out of sync Teaming can experience problems. To sync the adaptors

i) Disable teaming. Disable the second adaptor.

ii) At the command prompt

ipconfig 0 /registerdns

iii) Now enable the second adaptor and at the cmd prompt

ipconfig 1 /registerdns

iv) Re-enable teaming. See if this helps.


Using /registerdns will initiates manual dynamic registration for the DNS names and IP addresses that are configured at a computer.

see http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ipconfig.mspx?mfr=true

Good luck.

Rob

[Jeffry A Spain]

Thanks for your advice, Rob. I will try statically configuring speed, duplex, and link aggregation to see if that helps.

As I look at the situation further, I don't know that delaying my services is the complete answer. I also see errors in the event log relating to the Netlogon service and to the application of group policy.

I have found one interesting article on Google from Cisco. See http://www.cisco.com/univercd/cc/td/doc/product/voice/iptel_os/driver/hp_team7.htm#wp48758. The document, which relates to an earlier version of NIC teaming for Windows 2000, describes my situation and refers to a Teaming-CDP.cmd script to fix it. I haven't been able to find a copy of this yet, and wonder if you have ever heard of it?

Jeff.

[Robert S. Carr]

Jeff:

Don't forget the static IP as well. This actually takes the most time especially when using a Windows server for DHCP services. Switch/router based DHCP servers are typically much faster.

The errors you see could be related to teaming but to check, erase the log, diable teaming, reboot and note if they come back again.

I read through the article on Cisco. Hard to say if it would work. It seems to come with Cisco's software media server and is a file related to fixing a problem with the Services Control Manager spitting out errors. Are you getting the error 'HP Network Configuration Utility 7 service failed to start'?

Rob

[Jeffry A Spain]

Rob: I tested one of the servers, a DL360 G2, with the eight combinations of speed/duplex settings 100/full or auto, LACP negotiation (configured on the Catalyst 6500 switch) off or on (active), and IP address DHCP or static.

To avoid the having the services with domain user credentials fail, I had to set either a static IP address or fixed 100/full speed/duplex settings or both.

With LACP negotiation off, the NIC Teaming driver generated some warnings in the event log, but this did not result in errors in any other services.

For my purposes, setting the speed/duplex to fixed settings and leaving the IP address dynamic (we use DHCP reservations) will probably work the best.

I also spoke to HP support (case 3214364324). According to them, the Teaming-CDP.cmd script is proprietary to Cisco and their Media Convergence Server product, and thus not available to me. What it does remains an unknown.

Thus I believe I have a satisfactory resolution to this issue and a better understanding of the factors involved. I appreciate your input. Thanks. Jeff.

[Jeffry A Spain]

One other thing, when testing a DL380 G4 (much faster than the DL360 G2), I found that I had to set both a fixed link speed and duplex and a static IP address to get NIC teaming to work properly.

A caveat related to this: when changing from DHCP to static addressing, you have to delete the DNS A record for the target server and then execute ipconfig /registerdns on that server.

The reason for this is that with DHCP in use, the DHCP server does the initial registration of the target server with DNS and so causes the target server's A record to be created. The ACL on that A record contains the name of the DHCP server and not the target server. Thus when the target server is changed over to static addressing, the target server cannot update its own DNS record. Dnsapi warning 11166 appears in the system event log under this circumstance. By deleting the DNS A record and having the target server to recreate it with ipconfig /registerdns, the new DNS A record's ACL contains the name of the target server. Although I did not test it, the same problem would probably occur when changing from static addressing to DHCP, and would be solved by deleting the DNS A record. Note that associated pointer records are also deleted when A records are deleted.

[Lucky Luciano]

there are few options also for teaming on HP, for me was working in the same situation teaming based on MAC address, never on IP address, switch assisted load balancing.
On switch I've grouped both ports in same admin group, but oposit to recomended I've let switch to have speed/duplex - auto and same on server