Veritas Netbackup Performance Problems HP-UX 11.0

[James Bankston]

I am having terrible backup performance problems using Veritas Netbackup 5.1 on only my HP-UX Servers. Does anyone have any suggestions on some things I can try? I have verified that both servers at 100 Full Duplex.

[Steven E. Protter]

How much data are we backing up and how much time does it take.

Veritas can also back up via the SAN in you have one and that can improve performance.

Network infrastructure, heavy traffice from other users all can add to poor performance.

To find the cure, you need to track down the cause.

I found SAN worked best, followed by setting up a private network for backups on the servers secondary network.

I suggest looking at the logs of the system, network switch, veritas and other infrastructure for clues.

A network sniffer can catch packet collision.

SEP

[James Bankston]

We currently do not have a SAN solution in place. It 8 Hours to backup 72 gigs. The backup network is on its seperate network. We have some Sun Servers and none of them experience this problem.

[Jeff Schussele]

Hi James,

You probably need to have the NetBackup server call for more threads when it signals the client to backup.
This is an entirely server-driven option. You can't configure this on the client.
I have some heavily loaded large data-set servers that I need 8 threads to backup & it still takes 3+ hours.
Also I might suggest that IF the payload is that large you might consider using separate HBAs to "feed" the backup - i.e. don't use the same HBAs that the app uses.

My 2 cents,
Jeff

[yut]

backup via network is slow.try to backup via fc cable and make server that you want to backup as media server.


-yut-

[Yosef Rosenblatt_1]

Mr. Bankston,

The fact that your Sun boxes don't have any problems goes a long way to indicate that the issue is not the NB server. Here are some issues that may be part of the problem:

Check your kernel configurations, especially the memory configurations. Veritas's minimum configuration does not take into account any other software running that may need memory, e.g. performance monitoring software.

Do you have sufficient swap space?

To what verbosity level are your logs set?

While the following items may not be root causes they may help alleviate the symptoms.

You may want to look at your NetBackup policy:
Are you multiplexing?
Are you multistreaming?
How do you have your file systems broken up?

HTH,
Yosef

[Russ Hancock_1]

James,

We have similar performance on our HPUX v11 server, we currently backup 250Gb in around 14hrs... Quite dissapointing as fbackup is almost as quick.

Our Netbackup used to be even slower than this, and infact time out and never complete.

The main cause of slow performance in our case was related to the huge number of small files in the directories being backed up.

And a regular defrag of the OS also helped.

Good luck
Russ

[Steve Lewis]

We had performance problems at first and these items may help you find the issues.

We had the bp.conf file in the wrong directory - it must be located in /usr/openv/netbackup to be used.

We had a name resolution issue - some of the configuration files used host names on the slow lan, not the dedicated backup lan which caused the data to go down the wrong route. This needs to be set-up both from clients to server and vice versa and you must monitor netstat -i to be sure you get the right one.

The drives on the backup server had the wrong device file minor number. The filename is irrelevent but the minor number is interpreted by the driver to enable/disable hardware compression.

Check that your lan interfaces are configured the same as the switch ports. If one end is on autonegotiate and the other is hard-configured then you WILL get a duplex mismatch and the data will crawl. Checking the server configs is only half the story.

Check the amount of memory on the netbackup server - the more the better. Monitor the actual memory and swap usage during the backup because Veritas does use it.

100base is still slow for a backup. Consider APA on the backup server and preferrably 1000-baseT/S.

[sparky_2]

Hi James. Some things to check -
- verify that you are not using client compression within the backup policy. This often results in a slower backup rate.
- verify that your NIC is set to full-duplex and that auto-negotiate is switched off.
- check you are using Veritas recommended kernel settings. You should be able to download an up-to-date document detailing these from support.veritas.com

You can do various tests to identify whether the problem resides with NetBackup itself - eg. compare the time it takes to backup a directory using tar. You could also try using a disk storage unit if possible to check whether this encounters similar problems.

It may also be worth looking into using multiple data streams and / or multiplexing.

[rick jones]

If netbackup uses only one TCP connection per client, APA/trunking on the backup server may help the aggregate, but any one client will still be limited by the performance of a single link.

If lanadmin shows full duplex, but FCS errors, likely there is a duplex mismatch with the switch. If it shows half-duplex and _late_ collisions, it can indicate the same thing.

If there is a duplex mismatch, that means lost packets, which means that netstat -p tcp should show retransmissions. While a backup is in progress, consider a netstat -p tcp > before; sleep 60; netstat -p tcp > after and then run the two files through beforeafter (ftp.cup.hp.com dist/networking/tools) to see what the retransmission rate happens to be.

Might also check the TCP window size being used for the transfers. If there is any non-trivial latency you may need to increase window size. Netperf -t TCP_STREAM tests between a client and the backup server may help there.