NetBackup running VERY SLOW

[Joel Shank]

I am using Veritas NetBackup to backup my UNIX environment (HP and some AIX). Everything runs fine except for 1 AIX server and 1 HP server. They are configured identically to all the others as far as Netbackup is concerned. But when these servers perform FULL backups, they transfer the data at around 20 MB/sec. All the other servers run at better that 1000 MB/sec! I can not figure out why there is such a hugh difference here. All the servers transfer their data over a GB Ethernet, though their LAN cards are running 100 FULL DUPLEX (but so are all the others).

Veritas is no help. They told me about setting NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS, but these settings had no effect on these servers (but did slightly improve the others).

These two servers are no where near the size (data wise) than most of the others, so it isn't the amount of data. They don't run at the same time, and don't run concurrently with other large backups.

Has anyone had a similar problem? If so, how was it solved? What else can I look at? I have already used glance, netstat and lanadmin which don't show any problems.

Thanks in advance for your help :-)
jls

[Tim Sanko]

We are using Veritas 4.5 for our non-production backups.

We found that our slow servers were on subnets that also had Microsoft servers running with Microsoft networking active.

Our fix, private VLAN for BACKUP, and for Microsoft servers not to "share files and folders " on a NT oracle server.

Tim

[Tim Sanko]

There is also a hardware investigation you can do
1. Use lanadmin.
2. set ppa
3. display

At this point you can clear all statistics going forward after you look for teltale signs such as:
Inbound Errors = XXX
OutboundErrors = ZZZ

This was why we were slow.

Inbound Unknown Potocols = 2234520

[Steven E. Protter]

Few things to check.

If your switch is sciso change it from autonegotiate on the HP's port to manual, 100 BaseT full duplex.

Boot your HP box.(optinal)

lanadmin -x lan#

replace # with the number of the NIC card in question.

If you don't get 100 Base T full duplex, hard code it.

/etc/rc.config.d/hpbtlanconf

Now you have to boot your machine..

lanadmin -x lan#

One problem solved and future potential Ignite problems avoided.

Next: do you have a fiber cared on your Unix boxes(ie SAN environment)? If so, they can be configured onto the SAN and you can bump your speed up by a factor of five.

Veritas is much faster in such an environment.

While the backup is running, here is a diagnostic to help with the next step.

netstat -I lan# 3

You should see huge, steady transfer rates with this command/

Replace the # with the lan number.

SEP

[Joel Shank]

Thanks, Tim

The UNIX and NT networks are on different subnets, so I don't think they are causing any problem.

I am seeing no in or out errors, but I am getting Inbound Unknown Protocols (3146315). I have cleared the counters and see that this is accumulating.

What can I do about these Unknown Protocols? How did you fix it? What does it mean, anyway?

Thnx
jls

[Joel Shank]

Thanks, Steve

My lan card is already set at 100 FULL DUPLEX.

I ran the netstat -I lan0 3 command and see a steady rate. What is "huge". My numbers are anywhere from 200 to 500 packets (in and out and in total in and out). Is that good? How can I make it run faster?

I can't reboot these servers very easily as they are production.

Thanks,
jls

[Bill Hassell · Accepted solution]

Perform a large ftp transfer between the two boxes, perhaps 200-500 megs for one file. Repeat 3-4 times in both directions. ftp is the fastest protocol for network transfers and gives consistent results. If the ftp rate is significantly faster, then it's back to Veritas for better answers. Glance should show 1000's packets/sec for a 100Mbit LAN that is truly busy.

[Joel Shank]

Thanks, Bill

That was a great idea to do the ftp. I see now that it seems to be my server. The ftp of a 200 meg file is still running. It should have been done a while ago.

How can I find out why it is running so slowly? There is nothing different from the other servers that run fine! Is there some log I can look at or turn on?

jls

[Tim Sanko]

Joel,

Unknown protocols are often traffic that can be from routers, hub management interfaces, etc. Sometimes they are from Novell, or Microsoft hosts. In particular I believe netbios is an unknown protocol. Is it possible that you have a PC on your server network?

There are no errors I take it at the ethernet card, Did you check the switch/HUB port?

I hope that is a further help...

Tim

[Bill Hassell]

Run lanadmin and reset the stats. Then transfer the file. A switched network (no hubs) should be very clean with no errors and ideally, no unknown protocols. If you are using hubs, I would dump all of them. 100Mbit switches are very cheap now ($100 for a 16 port Linksys) and even feature auto-sense for cabling (no swapper cables needed).

[Steven E. Protter]

Answer to your question, what is huge.

I've seen numbers over 5000 during Ignite transfers.

Here is my definition of HUGE.

packets packets packets packets
629596316 1046948875 629832342 1047213197
3407 23828 3407 23828
3710 24155 3710 24155
3926 25535 3926 25535
4304 24351 4304 24351
3678 23755 3678 23755
3999 25749 3999 25749

You should do pretty well in a switched environment. If not, what I posted above might be a valid investigation path.

Apologies for the delay in answering your query. I never should have used the word huge.

The switch/router type I mis-typed is Cisco, just in case that gibberish was not understood.

SEP

[Michael Steele_2]

1) 'autosense' is not recommended for NIC???s instead use full duplex: 100FD
2) Check for and remove the ???LIMIT_BANDWIDTH??? in bp.conf
3) Remove the ???client compression??? argument in the backup policy
4) Check for fragmentation in the file systems that you???re backing up. Online JFS uses: fsadm ???F vxfs ???d ???D ???e ???E /mount_point
5) Consider Flashbackup utility
6) Full backups are fastest to restore but longest to backup
7) Incremental backups are slowest to restore but fastest to backup
8) Consider multiple data streams, AKA parallel backups

[Denver Osborn]

for what it's worth... I've run into the same type of issue w/ veritas netbackup. my prob was caused by mismatch of nic and the port settings of switch. you hadn't mentioned much about the network equipment involved, but if it's a managed switch, check the port's config... if possible you could move the box to another port on the switch where it's known good. or maybe set the port on switch to 100FD and the nic to 100FD rather than auto-negotiate.

Hope this helps,
-denver