strange network problem...

[BOUR]

We have a strange network problem between 2 different server rooms :
Server room A containing our Production servers with different type of Servers (N,L,K), different type of OS (HPUX 11, 11i)different type of network card ( all in 100 FD).
Server room B containing our Tests servers with different type of servers (K,D,L), different type of OS (HPUX 11, 11i), different type of network card ( all in 100 FD)
There are problems of Data transfert from different type (ftp, HP Omniback,..) between server from A to server from B for some test servers and server from B to server from A for others test servers ---> but not always !!! One time the line is ok ( more than 8000 k/s) one time really bad (10 k/s) and not at the same time for all servers.
All these transferts have the same routes ( same traceroute without problems).
Test from 100 HD, FD have been done between servers and switches.... Nothing !
Any ideas ????

[Massimo Bianchi]

Hi,
is all the equipment with fixed speed or os there any autosense device ?

They can impair performance very much.

HTH,
Massimo

[Stefan Farrelly]

Sounds like a speed/auto mismatch problem.

Confirm what your network settings are on all your HP servers using; lanadmin -x speed
where nmid is the ppa number of your card (usually 0 or 1). This is the only way to be absolutely sure what the current setting is (it may differ from your driver config file!)

Next, confirm on your switch exactly what each port is set to (using a command on the switch), ie. autonegotiation, 100Fd or whatever. They must all be the same.

We just had a problem like this recently and once we confirmed our Gigabit card was set to Auto on the HP (can only be set to auto for 1000FD), even though our network guys swore black and blue their switch port as also auto I finally found from the switch/card manufacturers that the switch port was only capable of 1000FD (ie, not auto). The only thing it was autonegotiating was flow control, not speed. We had to replace the switch card for one that was capable of auto to match the HP.

Also check syslog on each HP server to see if any auto messages from the network cards (search for auto, 100, duplex etc.) This will confirm a speed/auto mismatch.

[BOUR]

Servers : all forced to 100 FD AUTO OFF( in /etc/rc.config.d/*conf )and verified by lanadmin command.

Switches : All corresponding ports are in 100 FD AUTO OFF

[Stefan Farrelly]

A couple more ideas;

1. is flow control on each card and switch port ? you can confirm on the HP with;
lanadmin -x fctrl

2. Sounds like it could be an omniback problem. Do you have the latest omniback patches installed ?

3. Are all your test and prod servers using the same default gateway and dns servers ?

4. have you checked your lan cards for any errors (lanadmin).

[BOUR]

1. is flow control on each card and switch port ? you can confirm on the HP with;
lanadmin -x fctrl
---->> I don't know this flow control options, but when I performed your command line, it is returned : 100 FD auto, so...

2. Sounds like it could be an omniback problem. Do you have the latest omniback patches installed ?
----> not only Omniback data transfert problem : same with different type of ftp (standard, proftp)

3. Are all your test and prod servers using the same default gateway and dns servers ?

--->>same default gateway for all tests servers in server room B, DNS not used

4. have you checked your lan cards for any errors (lanadmin).

--->> lanadmin results for 2 test servers

SERVER X
PPA Number = 0
Description = lan0 Hewlett-Packard 10/100 TX Full-Duplex Manual TT = 1500
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 100000000
Station Address = 0x1083368110
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 8374
Inbound Octets = 3721121321
Inbound Unicast Packets = 44809428
Inbound Non-Unicast Packets = 58262
Inbound Discards = 299
Inbound Errors = 0
Inbound Unknown Protocols = 252
Outbound Octets = 2589565100
Outbound Unicast Packets = 23189316
Outbound Non-Unicast Packets = 299
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367
Ethernet-like Statistics Group

Index = 2
Alignment Errors = 0
FCS Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

SERVER Y
PPA Number = 0
Description = lan0 Hewlett-Packard 10/100 TX Full-Duplex Manual TT = 1500
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 100000000
Station Address = 0x306e0696e6
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 472871632
Inbound Octets = 4271706349
Inbound Unicast Packets = 12694982
Inbound Non-Unicast Packets = 896145
Inbound Discards = 4599
Inbound Errors = 0
Inbound Unknown Protocols = 740
Outbound Octets = 662482063
Outbound Unicast Packets = 11719990
Outbound Non-Unicast Packets = 4599
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367

Ethernet-like Statistics Group

Index = 1
Alignment Errors = 0
FCS Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

ANOTHER INFO given by Network team :

perhaps an uncompatibility problem between HP NIC and Catalyst 2948L3 .....

[Massimo Bianchi]

Did you try with a network analyzer, to see if there is traffic outside your knoledge ?

May be there are huge mass transfer of files or exports to nfs.

HTH,
Massimo

[BOUR]

we have from Unix side start a trace in both server rooms with nettl during a bad ftp transfert and in the same time network team sniffed the same data transfert from both side :
Experts alarm coming after analyze --->
- Loops on Same Request
- Idle Too Long
- Fast Retransmission
- Retransmission

[Stefan Farrelly]

If its affecting more than omniback then it is a network compatibility problem.

Check if flow control is on the HP servers with;

lanadmin -x fctrl

Check it on the switches too so they match.

Check the MTU size using;

lanadmin
lan
display

And on the first output it will show the MTU size on the HP card. Should be the same on all HP cards.

Check the card info on the HP lan cards with;

lanadmin -x card_info

Ticks and buf descriptors should be the same on all.

[BOUR]

We have during the bad ftp transfert an increase of FCS and inbounds Errors (lanadmin)?
What does it mean ?

[Stefan Farrelly]

FCS errors mean you have a network speed mismatch. Despite your earlier response that everything was set the same on the HP's and the switch I dont believe thats correct. Check them thoroughly yourself. I had the same problem recently, my networks guys swore that the settings they told me were correct, only for me to find out myself later that they were not.

[Andy Liu]

1. You need to verify the network device port speed configuration, recommend you
fix the 100 FD for port setting.

2. Maybe you need to set two
sniffer probe to get network
data, because some UDP
servieces will effect the
network performance. Check
your internal DNS server
work or not, some name
resolved query problem will
effect FTP,telnet
performacne.

regards,

andy