DL 380 G3 - Networking Problems

[Chris Kranz]

Tearing my hair out with this...

I have a brand new DL-380 G3, and since taking it out of the box have had networking issues, which is why it still hasn't made it into production yet.

The machine is running Windows 2003 and Exchange 2003, both service pack 1. It's running nothing else, was installed from the HP CD's, and updated from the website. Everything is as up-to-date as I can get it.

It seems to intermitently drop the network connection. Can't put my finger on why, when, where. There's no real reason, the server isn't doing anything in particular at that time. As it's still 'in build', no-one is accessing the server other than myself.

I've tried teaming the network cards,
i've tried configuring them seperately,
i've tried plugging them into different switches,
i bought a brand new gigabit switch which they are both currently plugged into,
i bought a d-link gigabit card and get the same problems with that, a little more rarely, but the problem is still there,
i've tried disabling each card seperately,
i've had them in 100 and 1Gb networks,
i've tried fixing the speeds 1Gb Full Duplex,

I've run out of ideas now, i run a ping constantly from my machine on the 3 IP address' it now has (including the d-link card) and intermitently it'll drop any one of the connections whenever it feels like it.

my machine is plugged into the gigabit network, but i know the problem isn't this end.

can anyone give me any help what-so-ever? i've also logged a support call which i'm waiting to hear back from... I'm starting to get pressure to get this machine into production as we're paying a lot of money for POP boxes, when we should be using this Exchange machine, but I can't launch it until this problem is solved...

please help!!!

[Chris Kranz]

... on a side note, I have 2 other DL380's, one of them an identical build, plugged into the same gigabit switch, and they don't suffer from this problem.

[Paul W_1]

I would presume you are using the onboard NC7781 nics. If so, read this advisory first.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=PSD_CO040408_CW01


Paul

[Rob Leadbeater]

Hi Chris,

What sort of switches are you using ?
Do you have any way of seeing errors on them - or confirm that the switch is indeed syncing up at 1GB Full Duplex.

Have you double checked the cabling ?

I'm also curious when you say "brand new" DL380-G3. The G3 hasn't been new for some time now...

Cheers,
Rob

[Stephen Kebbell]

Hi,

Just a shot in the dark here, but would this MS-KB article have anything to do with your problem?
http://support.microsoft.com/kb/898060/

I would also make sure you have the latest ROM and firmware versions installed.

Regards,
Stephen

[Chris Kranz]

-rob-

i'm using all 3com switches, i've tried it in both managed 10/100 switches, and the current unmanaged gigabit switch. the flashing lights on the front show it's working in full duplex 1Gb. I've also tried fixing the card (which i know still auto-neg's, but it's worth a try).

all cables are brand new cat6, straight out of the packet. the chances of all 3 new cables going to each of the cards are slim, but the cables have been swapped around between the machines.

'brand new' for me, it's not reconditioned is what i meant.

-stephen-

i've patched the box as much as i can see is available, and the MS-KB article you provided has this at the bottom...

"Note This security update is not applicable to Windows Server 2003 with Service Pack 1."

... which is exactly what i'm running. it confuses me as it's exactly the same build (as far as i can tell) as another server we have which is running perfectly, and is our SQL server.

-paul-

i'll have a look at this, i'm using driver version 7.103.0.0 (27/01/2005), and the server is far from under load. it's an exchange box where the only traffic to it is myself, occasionally...

[Chris Kranz]

many thanks for your help so far!!!

[Chris Kranz]

just installed cp005567, updating the driver to 8.39.1.0...

the problem stopped for about a minute, i got excited, then pings started to get dropped again... the cards still seem to have intermittent problems with connection

[Rob Leadbeater]

Hi Chris,

Can you confirm what the current NIC configuration is - ie whats cabled to what, what subnets things are on etc...

Cheers,
Rob

[Chris Kranz]

rob...

currently it's something like this...

I have a 3com gigabit switch, unmanaged (i can go grab the model number if required), all the servers and subsequent switches plug into this. The clients plug into 3com switches (10/100) with gigabit uplinks. My personal machine has a gigabit card and is plugged in next to the server in question.

All cabling is brand new CAT-6 cables.

The onboard cards are HP NC7781 with driver 8.39.1.0 installed. These are both plugged into the same switch, with the IP address of .146 and .147 on the same subnet of 255.255.255.0 both have the same gateway, dns settings, everything. nothing special setup here. netbios is disabled on all cards (no pre-2000 machines here).

i know that this can cause some windows conflicts, so i have tried disabling windows networking on 1 of the cards, even tried removing the gateway from one of the cards. this made no difference, so everything is back to default.

there's no vlan's setup, everything is very ordinary and boring. maybe it's not ideal, but before i get advanced, i'd like to get the basic setup working!!!

[Rob Leadbeater]

OK... It sounds like something may be getting confused with routing - although you should have ruled that out by disabling one of the NICs...

If you're PC is on the same subnet as the server:
Remove the default gateway from all devices, including your PC, and then try pinging the various IP addresses.
That should at least rule out any possibility of a packet being received on one interface and going out on another.

[Chris Kranz]

okay, changed this on both the server (where i've kept the setting changed for now) and my machine. obviously had to change it back to get back on here, but still having the problems...

i have a constant ping running from my machine (using fping) to multiple machines. the 3 IP address' of this machine, plus another 2 DL380's we have on the network (with teamed network cards) and a filer (again with a teamed network card). all machines are plugged into the same gigabit switch, the HP in question is the only one with problems... and they are stupidly intermittent. right now all cards are responding, 30 seconds ago the 2 internals were failing. 5 minutes ago 1 internal and the d-link were failing...

[Rob Leadbeater]

> ... on a side note, I have 2 other DL380's,
> one of them an identical build, plugged into
> the same gigabit switch, and they don't suffer
> from this problem.

That identical build wasn't done by a cloning process was it by any chance...?

I'm just wondering whether you've managed to get a duplicate MAC address on your network somewhere which is getting your switches confused - I've seen this before when cloning machines that have been configured with HP Teaming...

If you can get onto a managed switch you might want to check out its arp table to make sure that the MAC addresses are pointing to the IP addresses you're expecting...

Cheers,
Rob

[Chris Kranz]

sorry, identical build as in hardware. the machines are both windows 2003, but one is the domain controller and sql server, the other (my problem machine), is an exchange server in the making...

i had a problem with the mac address' with the machine when i first got it. i used the HP tool to team the 2 NIC cards, and when i disolved it, it crashed and left both cards with the same MAC address. A complete re-install fixed this. But again, this shouldn't be the problem as i've run the machine on 1 network card, with the other both disabled and unplugged.

[Chris Kranz]

don't you hate it when you ask everyone for advice and help, then stumble onto your own solution???

so anyway...

I noticed the arp tables on my machine weren't quite correct, so I cleared them down... I then disabled the d-link card on the server, and resetup the team on the internal NIC cards and gave it a new MAC address, separate from the ones on either card. The machine now 'seems' to be running fine and returning all pings I send to it.

I think what happened was when i setup the original teaming, it clones the MAC address of the first card, combined with plugging the network card into varying switches and trying different configurations with the switches, the ARP tables had got confused. Now that the new team has a different MAC address, it has cleaned up the ARP tables and things seem to be running fine.

I'm going to closely monitor this and make sure it stays working...

*fingers crossed*

many thanks for everyone's help, i hope my problems here help other people with these kind of issues...

[Chris Kranz]

A week later, and it looks like the problem is more a network issue.

The DL380 in question is performing well now, however the network issues are still there.

I'll randomly lose connection to the DL380 or to my NetApp Filer at any point, randomly, but solid for about 5 minutes, then it'll work again.

I've tracked this back, and during the downtime, if I fire up my arp tables from my machine (not the server) it has the wrong MAC address for the down servers, infact it has the MAC address of the gateway (Cisco PIX). If I clear my arp tables, then connection is regained immediately.

The connection loss also isn't on the server, it's on the client, and not fixed to just my machine, any machine can and will lose connection at any point.

Not even sure where to go in trouble shooting this now. I appreciate it's probably not an HP problem, thankfully I've ruled that out, but there has to be away of discovering the root of this problem?