suggestions to improve network IO

[Arnold_14]

Hi all,

I have a database server that's running HP-UX 11.11. It has 4 CPUs and 4G of RAM. This is a pretty new server.

This server has several lan interfaces and one of them is a Gig-E card that is dedicated for backup purposes.

I've just installed GlancePlus to monitor the system, and for the past few days, it's been complaining that the Network Bottleneck = 60%, sometimes it goes up to 95% and it becomes an Alarm. This is for that Gig-E card when the backup is running.

From netstat, everything looks OK. These numbers stay constant most of the time when the backup is running.

In Pkt Rate: 2323
Out Pkt Rate: 4560
Coll Rate 0
Err Rate 0
Input Pkt 31625
Output Pkt 62167
Coll 0
Err 0
Speed 1000000000
AutoNeg: One

The problem is, the backup never finishes in the 6-hour backup window. Somehow, it's running very-very slow. The CPU. Memory, and Disk IO, are in a good shape according to GlancePlus. The Disk I/O sometimes hits 95% activity for about a few seconds every half an hour or so.

Anything I can do here? Those numbers look OK to me (coll and err are 0s).

Thanks a lot
Arnold

[Patrick Wallek]

How much data are you trying to back up? What type of tape drive are you backing up to? Just because you are going over Gig Ethernet doesn't mean that you will automatically get good performance. I am not really fond of network backups no matter what type of network you are going over.

[Bill Hassell · Accepted solution]

Since the LAN is dedicated, is there anything else connected to it? Glance has it's own idea about what is overloaded so I would not pay attention to the alarm, just work out the actual backup data rates. Start with the maximum throughput possible on the LAN. The data rate is 1Gbit/sec which roughly 100Mbytes/sec. You will almost never see that throughput rate. I would figure 50Mbytes/sec as a usable rate assuming no other activity. Since this is a backup, you would expect huge data rates.

At 50Mbytes/sec, 1 hour is 180Gbytes/hour which is about 1100Gbytes during the 6 hour window. So if you're trying to backup 2 terabytes, the data rate is far too low to fit into 6 hours. Note that modern tape drives can exceed LAN speeds. That's why LAN-based backups will use several LAN cards running at the same time, usually from different servers and the backup software collects all the data fast enough to keep the tape drive busy. For terabytes, fiber SAN is the only way to meet the throughput requirements. And of course, classic backup tools such as tar and cpio are not appropriate for 100's of Gbyte backups.

[Tim D Fulford]

The packet rates look low to me.. 4560 pkt/s at 1500 byte MTU (Max Transport Unit) is about 6.5 MB/s... which is about SCSI speeds? (You seem to have two Input Pkt figures, the seconds one I assume is cuumlative and not per sec)

I suspect if there is a bottleneck it could be the disks. If you have a wizzy SAN, XP?? or EMC then I will need to eat my words. But if it is, say, a couple of SCSI disks then I think the disks may be at issue.

Have you tried to use MeasureWare to monitor the backup window? These give 5 minute averages which smooths out the spikes you may see? The other thing with measure ware is you can see how the disks are performing, their services times & disks queues etc.

good luck

Tim

[Arnold_14]

Yeah, I think there's something wrong with the SAN disks too. A few users have been complaining that there processes take forever to finish. GlancePlus returns 'Disk Bottleneck = 100%' a few times.

The SAN admin un-installed the EMC PowerPath software a while ago cause it was causing problems. But I guess now, we have to pay the price, slow performance from the SAN. I may have to push him to install the PowerPath back.

Thanks
Arnold

[rick jones]

While Glance is _considerably_ better about networking bottlenecks than it used to be, it is still a bit, well, conservative I suppose when sounding an alarm. Likely as not, you can safely ignore network bottleneck warnings from glance.

The important thing to look for are packet drops (collisions are not drops, _unless_ it is a _late_ collision) and retransmissions by the ULP's (Upper Layer Protocols - eg TCP, NFS).

Might also look for zero-window probes - if the backup software is using the default socket buffer size (32768 bytes) it will _not_ achieve optimal performance over a GbE link.

If you want to check-out the link without disc I/O being an issue, you might consider using netperf - http://www.netperf.org/ and the TCP_STREAM test.

[Ted Buis]

If you are no longer using PowerPath, make sure that you revert back to PVlinks, and put half of the disks (estimated activity) as primary over path A and the other half of the disks as primary over path B. It isn't dynamic load balancing but usually is works well.

On the original subject, I saw a test done by a customer a few years ago with some of the original GigE cards on a V-class get up to 55 MBytes/sec using Veritas NetBackup with 4 streams, but only 30MB/sec with a single stream. This points out the necessity of combining multiple sources to keep the GigE pipeline at anywhere near peak performance.

[Steven E. Protter]

Thinking outside of HP box.

Switch port configuration. It might be hard coded to 100 BaseT manual or something. That might be bothering Glance.

Do a big file transfer and see what rate in MB/second you get. In our current configuration, we get around 9 MB/s in prime time.

SEP