1000Base-T problem rx6600

[Generic User OI IIS]

Hi Admins,
We have several rx6600 servers with 1000Base-T network interface cards. ("HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core" and "HP A7012-60601 PCI/PCI-X 1000Base-T Dual-port Adapter") These NICs are configured as auto-neg, and are running as 1000 Full Duplex, using iether driver.
It seems to work fine, however when we try to copy large files over the network, after some time (somewhere between 4 and 12 Gb) the copy hangs, and will not continue anymore. When I enter a bdf command, it hangs as well. Some other commands, like ps, work just fine.
Copying the same file from a rx2660 to another rx2660 works fine. These NICs use the igelan driver, but copying the same file to an rx6600 results in the hang.
We tried to use a crosscable between a rx6600 and a rx2660, same result.
We installed patch PHNE_36236, for this could solve some issues with high loads, but problem remained.
Has anyone any idea, what we could try?
Thanks in advance,
Addie Wevers.

[Steven E. Protter]

Shalom,

Install the latest QPK bi-annual patch and HWE, Hardware Enablement patch on the system.

Then re-test.

Then check that network switch port settings, hardware, ios and such are consistent when compared to the rx2660 systems.

SEP

[Duncan Edmonstone]

Addie,

First off - don't use a x-over cable with GbE - it's not supported.

Are you saying that with the exception of the different drivers these systems are patched up to the same level?

Have you looked to see if you are generating any significant errors on the rx6600 servers in lanadmin?

HTH

Duncan

[Generic User OI IIS]

Hi SEP and Duncan,
Thanks for your replies.
We installed the server only a few months ago, using the june 2007 media. When I have a look at HWE of december 2007, I can't see any patches that is related to our problem.
The ports on the switches are configured the same way as for the rx2660's.
When we run netstat during the rcp on a rx2660, (to another rx2660) we see consistent transmission of 443,000 packets (outgoing) every 10 seconds, and 35,000 incoming.
We used the crosscable for testing purposes only, trying to exclude the network.
Lanadmin reports 0 inbound- and outbound errors.
Regards,
Addie Wevers.

[Duncan Edmonstone]

Addie,

Of course the problem could be elsewhere than in the driver - have you tried using something other than rcp? Can you get the same issue with an FTP or copying onto a NFS mount?

HTH

Duncan

[Generic User OI IIS]

Hi Duncan,
We have just tried an ftp of the large file, and after 10Gb it stalled.

I also logged a software call at HP response center now.

Thanks again for your effort.
Addie

[Generic User OI IIS]

We have found out that the problem was not caused by the network card, but the cause has to be found on the server itself.

Copying a file over the network to /dev/null was extremely fast. (110Mbytes/sec)

However, copying the same file over the network to a file, resulted in a hang of the ftp-session. During that time bdf hang as well.

When we copied a large file from filesystem A to filesystem B, (both on the same server) it resulted in a hang of the cp command. During that time bdf hang as well.

When we stopped mirroring of the filesystem, the problem was gone!! Has anyone else ever seen this problem, and what to do about it?

Tia,
Addie.

[Pete Randall]

I have to wonder about your comment that bdf hangs as well. That's usually because an NFS mount has gone sour. To where are you copying this file? You say you're copying it to the rx6600, but where on the rx6600?


Pete

[Generic User OI IIS]

Hi Pete,
We have seen this on all kind of Integrity servers we have. (rx6600, rx3600, rx2660)

We copied a file from /var to the filesystem of /dev/vg01/lvol1 and back. We are not using NFS on any of the servers.

When the copy stalls, the bdf commands hangs as well.

Addie.

[Torsten.]

IMHO this sounds like a disk or file system problem, not like a network problem.
If the system can't read from the disk, it cannot transfer this data over the network.

You should check your disks.
Do you have a smartarray controller connected to the disks or do you use the integrated SAS controller (with integrated RAID)? I assume you have internal disks only.
If you post an "ioscan -fn" this would be clear.
Anything in syslog?

[Pete Randall]

Hi Addie,

OK, it doesn't necessarily have to be NFS at fault. As Torsten says, it could well be a disk or file system problem - some sort of I/O issue would cause the hang in both the copy and in bdf.

Do you have Glance? You could use Glance to look at the copy process and, hopefully, find out what it's waiting on.


Pete