BACKUP uses nearly 100% CPU

[Stephen Eickhoff_1]

BACKUP uses a lot of CPU on our Alphaserver DS25 with 2 GB RAM running 7.3-2. The documents that DEC/Compaq/HP put our regarding BACKUP performance don't seem to address high CPU usage. We are backing up to an SDLT320 with encryption. Any suggestions?

[Hoff]

If you're using the encryption mechanisms within BACKUP (which required an add-on back on OpenVMS Alpha V7.3-2, and became integrated in V8.3), that's certainly a compute-intensive process.

What is the BACKUP command you're using?

Are you using host-based encryption, or drive-based encryption?

Have you monitored the OpenVMS Alpha box to see (for instance) the processor mode involved? High user-mode CPU usage within the process running BACKUP can be expected, but (for instance) high inner-mode time in BACKUP or processor usage arising within another process running in parallel to BACKUP might point to something else arising here.

Are you current on your ECO kits?

[Stephen Eickhoff_1]

We are using the host-based encryption. We are not using compression on the drive.

The command is:
BACKUP/IMAGE/LIST=DISK2BU.LST/IGN=(INTER,LABEL)/ENCRYPT=(NAME=AFTBUKEY)/BLOCK_SIZE=65535 DISK$DISK2: MM1:DISK2.BKP

These are the latest patches:
DEC AXPVMS TCPIP_ECO V5.4-156 Patch Install 26-JUL-2009
DEC AXPVMS VMS732_F11X V6.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_NETACP V1.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_SYS V17.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V17.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V16.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V15.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_PCSI V5.0 Patch Install 12-JUN-2009

[Hoff]

>We are using the host-based encryption. We are not using compression on the drive.

FWIW, data is generally compressed before it is encrypted. Reliably-encrypted data cannot be compressed. (The repeating patterns within data that the compression software is based upon are exactly what the encryption algorithms look for and seek to eliminate.)

The obvious suspect here is the overhead of encryption.

This suspicion usually easily verified, too. Shut it off for a BACKUP test; observe the difference from a BACKUP with and without encryption enabled.

Testing with the available AES-based encryption in V8.3 would be an interesting comparison, too.

Options here can potentially include an encrypting drive and there are a few of these on the market; offloading the overhead to a dedicated engine.

[Andy Bustamante]

Using the ENCRYPT layered product will consume a significant amount of CPU, that's the way it works. In our ES-40 and ES-47 systems, we see the bottleneck as CPU. We're also at 7.3-2.

You might consider moving to OpenVMS 8.3. There are some improvements reported in Encryption. Someone here may have a experience to report on.

Andy Bustamante

[John Gillings]

> BACKUP uses a lot of CPU

Excellent! You want BACKUP to run fast. It's designed to use as much resource as you will allow it to get the job done quickly.

You have 3 basic resources on your system: CPU, memory and I/O bandwidth. BACKUP gives you 3 knobs to adjust resource consumption.

CPU is controlled by the CPU priority of the process running BACKUP. If you want it to use less (or, rather, let other processes take priority), reduce the priority of the BACKUP process, or increase that of other processes.

Memory is controlled by the WSQUOTA of the BACKUP process.

I/O Bandwidth is controlled by the FILLM of the backup process.

All other quotas should be relatively "infinite" to these three (see tables in the BACKUP manual for deriving workable values relative to WSQUOTA and FILLM).

Encryption adds a CPU burden to the process. The real question you need to answer is if you need the CPU for something else. If not, you paid for it, so using it shouldn't be an issue.

[Stephen Eickhoff_1]

This DS25 has the fastest CPU in the datacenter; however, while the other systems are ES40s with slower CPUs, I think they all might have two. That could explain why they aren't being bogged down.

To clarify, Hoff, we are NOT using compression. Historically, we never have, but with the addition of encryption we are conscious that it would actually have a negative impact.

[Andy Bustamante]

>>>This DS25 has the fastest CPU in the datacenter; however, while the other systems are ES40s with slower CPUs, I think they all might have two

$ show CPU

Will tell you how many CPUs you have installed in the DS25. It will support up to 2 CPUs installed. If you only have one CPU, a second is eaily added along with an SMP license. The ES-40 will support up to 4 CPUs. In a 4 CPU system, we see backup consuming one CPU with no noticeable user impact. This is in a clinical 7x24 system.

Andy Bustamante

[Cass Witkowski]

Hoff,

Would a block size of 61440 be better than 65535?

Also with the SDLT and Ultrium tape drives can we set /NOCRC and /GROUP=0

[Hoff]

>Would a block size of 61440 be better than 65535?

Try it.

I'm aware of various theories here, and empirical tests always trump those. One of the common values floating around is 32256; that's what I have tended to use. I'd not tend to expect a huge difference from 32256 up to the limit, either.

I've had much better results with better targeting the data covered by the backups, moving to faster hardware, and particularly with knobs such as compression (yep; before an encryption) to reduce the amount headed to the drive, and /NOALIAS and such. With moving volatile data off the system disk and then archiving that disk once a quarter or after a significant change or upgrade or ECO or such. With sending less data out to the media.

The other tuning target here is being able to feed the data into to the drives; shoveling the bits through BACKUP, and (more importantly) feeding the buffers in the streamer and avoiding data starvation and thus keeping the drives themselves streaming at speed. Starved/stalled tape drives have to re-position and restart.) Also look at your interconnect speeds and device speeds. The FC SAN 2Gb gear is slower than some of the newer interconnects, newer LTO and Ultrium is faster than old (presuming you can keep the maw fed with data), and a fragmented drive is always a problem.

>Also with the SDLT and Ultrium tape drives can we set /NOCRC and /GROUP=0

That depends on how much you trust the tape and the tape ECC and the media itself, and considerations siuch as whether or not you can fit your archival processing into your available window.

I know folks that do trust the tape ECC. And I know folks that don't. I tend to have some trust in DLT and SDLT and such, and DDS makes an exceptionally good doorstop.

[a version of this has been posted to the web site.]