1752763 Members
4745 Online
108789 Solutions
New Discussion юеВ

Re: Poor performance

 
SOLVED
Go to solution
Bill Hassell
Honored Contributor

Re: Poor performance

60Gb per hour is impossible over a network. You are looking at the speed of the tape drive connected directly to a computer. Networks are VERY slow compared to today's tape drives. Figure less than 50% of the network's native speed is available for data (this accounts for overhead, packetizing, etc) and networks are rated in bits, not bytes. A best case formula would be: wire-speed / 20 = bytes/sec. So for 100BaseT, that's 5Mbytes/sec. So your stats:

> Backup Speed ....... 4238,18 (KB/s)

is right on for a 100Mbit link. You could try changing all the NICs and switches to 1000BaseT but now CPU/driver time may limit the maximum speed so don't expect 50Mbytes/sec, more like 30 or so. With 10000BaseT you may want to use jumbo frames but this may be incompatible with Windows.


Bill Hassell, sysadmin
Ted Buis
Honored Contributor

Re: Poor performance

Try an FTP of a file over your network. Assuming it shows you the transfer statistics as it does in UNIX, you can see the limitations of your network. I've done 7 or 8 MBytes/sec over FDDI at 100Mbits/sec, with FTP but Bill says 5MBytes/sec. That would be 18GBytes/hour best case. With backup, it isn't just reading the disk, but marking each file as to when it was last accessed, and there is file system overhead, so you never can get the highest speeds that you might think just in getting the data off the disk. Then there is the TCP/IP overhead. You still may need to combine streams just to get as much as you can over the network.
Mom 6
Eric_260
Frequent Advisor

Re: Poor performance

Yeah, right I was mixing up Mbps in MBps!
Damn it! I hate that speaking about Mbps and MBps there's always a mix up somewhere...

anyway, I know the implications of file access, open/close, TCP overhead etc...

But still its normal to get 4MB/s on a 100Mbps ?

Because okay let's put the network at 90% capacity. So 90Mbps.

90 / 8 = 11.25 MB/s
If I do a backup of a single filesystem, I never get that speed, always get 4MB/s.

That's only 32 Mbps. 32% of the network.
If I do a transfert between the 2 machines I get the 100% of the Network usage.

I know that Network are usually slow for backups, but that's the best option that my company have so far. So I need to maximize the network usage.

Anyway I'll try to get another solution in place. Because if 4MB/s is really the max I can get, I won't go anywhere. I guess that's it i'm at the maximum speed we can do.

Thanks for all your help!
Bill Hassell
Honored Contributor
Solution

Re: Poor performance

90/8 is not the right calculation. True, a byte is 8 bits but that does not account for the TCP/IP overhead. Let's start with an 8 Kbyte block. It can't be transmitted over TCP/IpP on a 100BaseT link. The maximum is only 1500 bytes per packet (rough numbers). So the 8Kb block must be sent as 6 packets. Each packet needs TCP/IP headers that include IP addresses (source and destination) along with other details. So that 8bit byte is looking more like 10bits with all the overhead. But the data doesn't travel in one direction. The packets must be disassembled, transmitted, acknowledged, retransmitted if necessary, and reassembled into the original data record. That's where server overhead (driver time) gets involved.

Secondly, you'll never get 90% throughput on your 100Mbit link. After you browse Rick Jones' NetPerf web pages, you'll understand more about real network performance. http://www.netperf.org/ is the place to go for LAN performance. With some TCP?IP tuning, you might improve on the 50% number but not by much.

Finally, as mentioned before, you'll probably get shoe-shine performance out of your tape drive and that will seriously affect throughput as well as prematurely wear out your tape drive. You pay a very stiff penalty for not keeping the drive busy. In the old days of reel-to-reel tapes, the drive actually started, recorded and stopped for every record (really old stuff). These drives would literally buzz as the capstan rollers and brakes did their work. Later, the drives started buffering a few Kbytes and allowed the tape to avoid the mechanical start/stop cycles and performance soared.

Then in the 80's, the concept of a streaming tape drive became a reality. No more start/stop. Instead, a small motor would slowly spin the tape up to full speed. Once at speed, data would be recorded continuously. The idea is that the computer would keep sending data fast enough to keep the tape moving. But if the computer could not keep the data coming fast enough, the tape drive would run out of data (an underrun condition) and it would have to stop, backup, take a running start again, figure out where it was on the tape, find the end of the last record and then start recording again at just the right spot.

Streamers were inexpensive nad performed well but were terrible if the data stream wasn't fast enough, and for an entire tape, you could resposition several thousand times thus causing recording head wear equivalent to several hundred tape passes without repositioning.

Now, virtually all tape drives are streamers. The difference is that they have a ENORMOUS appetite for data. The LTO-2 drive is no exception. True, these drives have bigger data buffers (many megs) but unless the data stream is slightly faster than the tape drive (measured over many seconds), the tape will still be repositioned once the buffers are exhausted. This is a case where a slower tape drive is actually a better choice. You match the average data rate of the tape drive with the data rate of your backup stream.

So the bottom line is that you need a very fast link between computers. GigE is a possibility, just computer-to-computer if necessary, or aggregate multiple LAN cards if both ends support it. Otherwise, you may need to look at a local tape drive to keep the expected backup speeds and not destroy your drive in 6 months.


Bill Hassell, sysadmin
Eric_260
Frequent Advisor

Re: Poor performance

6 Months, really ?

mmm, I think we will start working on getting
something running on GigE for the moment. That will be better.


Well, thanks to all who have provided some answers to help me stop mixing Mbps and MBps! ;)

And understanding how the drives works in streaming!

Thanks again!
Hein van den Heuvel
Honored Contributor

Re: Poor performance

The largsh number of files will also play a role. It will make it harder to keep the tape streaming. No chance to do very large IOs. 500K files in 16000 seconds is about 35 files/second. One would expect 2 or 3 IOs umper file (inode + data) so the disk probably was doing 100 IO/sec. That would be approaching the max, depending on exact placement. So do NOT expect a 10x speed bump if you get 10x more potential throughput from the wire. 2x maybe.

Hein.
Bill Hassell
Honored Contributor

Re: Poor performance

Hein is quite right about small files. sar -a 1 will show you how busy the directory gets during a backup. That's why fbackup and Data Protector launch multiple processes to grab data as fast as possible. If the disks are buffered with an array controller that has a lot (100's of megs) then multiple files can be opened and read at the same time. As a part of your planning process, you might borrow the Ultrium drive and attach it directly to the HP-UX machine. That will provide two important benchmarks: a full speed channel and whether small files are degrading the maximum speed of the tape drive.


Bill Hassell, sysadmin
Ted Buis
Honored Contributor

Re: Poor performance

In one reply Eric said that he was running DP on Windows, so unless he has a UNIX client, I don't think sar will help.
Mom 6
Eric_260
Frequent Advisor

Re: Poor performance

Yep, I'm in windows.

Anyway, so far I've been able to reduce the backup time of about 1 hour for my 6 backup groups.

I've attached the backup server in GigE to his switch, and the switch is connected to the main switch via GigE.

All the other servers are connected to their switch in 100MBps to their switch and then the switch to the main one.

(Yeah that's not the optimal setup, but that's what we have now)

At least now the backup server can receive a lot of stuff from other servers on the network.

So for now that's not so bad, but could be wayyyyy better! I'll have to deal with this, and I'll try to push to create a better infrastructure for the backups.

Thanks Everyone!
Bill Hassell
Honored Contributor

Re: Poor performance

I think Eric is running DataProtector tape server on Windows but the backup is coming from DP client on the HP-UX box. The reason to run sar -a 1 is to see how much directory activity goes on during the backup period.


Bill Hassell, sysadmin