topic Re: BACKUP uses nearly 100% CPU in Operating System - OpenVMS

BACKUP uses nearly 100% CPU

Stephen Eickhoff_1 — Tue, 17 Nov 2009 20:44:40 GMT

BACKUP uses a lot of CPU on our Alphaserver DS25 with 2 GB RAM running 7.3-2. The documents that DEC/Compaq/HP put our regarding BACKUP performance don't seem to address high CPU usage. We are backing up to an SDLT320 with encryption. Any suggestions?

Re: BACKUP uses nearly 100% CPU

Hoff — Tue, 17 Nov 2009 20:55:38 GMT

If you're using the encryption mechanisms within BACKUP (which required an add-on back on OpenVMS Alpha V7.3-2, and became integrated in V8.3), that's certainly a compute-intensive process.

What is the BACKUP command you're using?

Are you using host-based encryption, or drive-based encryption?

Have you monitored the OpenVMS Alpha box to see (for instance) the processor mode involved? High user-mode CPU usage within the process running BACKUP can be expected, but (for instance) high inner-mode time in BACKUP or processor usage arising within another process running in parallel to BACKUP might point to something else arising here.

Are you current on your ECO kits?

Re: BACKUP uses nearly 100% CPU

Stephen Eickhoff_1 — Tue, 17 Nov 2009 21:08:22 GMT

We are using the host-based encryption. We are not using compression on the drive.

The command is:
BACKUP/IMAGE/LIST=DISK2BU.LST/IGN=(INTER,LABEL)/ENCRYPT=(NAME=AFTBUKEY)/BLOCK_SIZE=65535 DISK$DISK2: MM1:DISK2.BKP

These are the latest patches:
DEC AXPVMS TCPIP_ECO V5.4-156 Patch Install 26-JUL-2009
DEC AXPVMS VMS732_F11X V6.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_NETACP V1.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_SYS V17.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V17.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V16.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_UPDATE V15.0 Patch Install 12-JUN-2009
DEC AXPVMS VMS732_PCSI V5.0 Patch Install 12-JUN-2009

Re: BACKUP uses nearly 100% CPU

Hoff — Tue, 17 Nov 2009 21:20:33 GMT

>We are using the host-based encryption. We are not using compression on the drive.

FWIW, data is generally compressed before it is encrypted. Reliably-encrypted data cannot be compressed. (The repeating patterns within data that the compression software is based upon are exactly what the encryption algorithms look for and seek to eliminate.)

The obvious suspect here is the overhead of encryption.

This suspicion usually easily verified, too. Shut it off for a BACKUP test; observe the difference from a BACKUP with and without encryption enabled.

Testing with the available AES-based encryption in V8.3 would be an interesting comparison, too.

Options here can potentially include an encrypting drive and there are a few of these on the market; offloading the overhead to a dedicated engine.

Re: BACKUP uses nearly 100% CPU

Andy Bustamante — Tue, 17 Nov 2009 22:15:35 GMT

Using the ENCRYPT layered product will consume a significant amount of CPU, that's the way it works. In our ES-40 and ES-47 systems, we see the bottleneck as CPU. We're also at 7.3-2.

You might consider moving to OpenVMS 8.3. There are some improvements reported in Encryption. Someone here may have a experience to report on.

Andy Bustamante

Re: BACKUP uses nearly 100% CPU

John Gillings — Tue, 17 Nov 2009 22:21:11 GMT

> BACKUP uses a lot of CPU

Excellent! You want BACKUP to run fast. It's designed to use as much resource as you will allow it to get the job done quickly.

You have 3 basic resources on your system: CPU, memory and I/O bandwidth. BACKUP gives you 3 knobs to adjust resource consumption.

CPU is controlled by the CPU priority of the process running BACKUP. If you want it to use less (or, rather, let other processes take priority), reduce the priority of the BACKUP process, or increase that of other processes.

Memory is controlled by the WSQUOTA of the BACKUP process.

I/O Bandwidth is controlled by the FILLM of the backup process.

All other quotas should be relatively "infinite" to these three (see tables in the BACKUP manual for deriving workable values relative to WSQUOTA and FILLM).

Encryption adds a CPU burden to the process. The real question you need to answer is if you need the CPU for something else. If not, you paid for it, so using it shouldn't be an issue.

Re: BACKUP uses nearly 100% CPU

Stephen Eickhoff_1 — Tue, 17 Nov 2009 22:23:47 GMT

This DS25 has the fastest CPU in the datacenter; however, while the other systems are ES40s with slower CPUs, I think they all might have two. That could explain why they aren't being bogged down.

To clarify, Hoff, we are NOT using compression. Historically, we never have, but with the addition of encryption we are conscious that it would actually have a negative impact.

Re: BACKUP uses nearly 100% CPU

Andy Bustamante — Tue, 17 Nov 2009 22:55:44 GMT

>>>This DS25 has the fastest CPU in the datacenter; however, while the other systems are ES40s with slower CPUs, I think they all might have two

$ show CPU

Will tell you how many CPUs you have installed in the DS25. It will support up to 2 CPUs installed. If you only have one CPU, a second is eaily added along with an SMP license. The ES-40 will support up to 4 CPUs. In a 4 CPU system, we see backup consuming one CPU with no noticeable user impact. This is in a clinical 7x24 system.

Andy Bustamante

Re: BACKUP uses nearly 100% CPU

Cass Witkowski — Wed, 18 Nov 2009 00:38:56 GMT

Hoff,

Would a block size of 61440 be better than 65535?

Also with the SDLT and Ultrium tape drives can we set /NOCRC and /GROUP=0

Re: BACKUP uses nearly 100% CPU

Hoff — Wed, 18 Nov 2009 04:02:49 GMT

>Would a block size of 61440 be better than 65535?

Try it.

I'm aware of various theories here, and empirical tests always trump those. One of the common values floating around is 32256; that's what I have tended to use. I'd not tend to expect a huge difference from 32256 up to the limit, either.

I've had much better results with better targeting the data covered by the backups, moving to faster hardware, and particularly with knobs such as compression (yep; before an encryption) to reduce the amount headed to the drive, and /NOALIAS and such. With moving volatile data off the system disk and then archiving that disk once a quarter or after a significant change or upgrade or ECO or such. With sending less data out to the media.

The other tuning target here is being able to feed the data into to the drives; shoveling the bits through BACKUP, and (more importantly) feeding the buffers in the streamer and avoiding data starvation and thus keeping the drives themselves streaming at speed. Starved/stalled tape drives have to re-position and restart.) Also look at your interconnect speeds and device speeds. The FC SAN 2Gb gear is slower than some of the newer interconnects, newer LTO and Ultrium is faster than old (presuming you can keep the maw fed with data), and a fragmented drive is always a problem.

>Also with the SDLT and Ultrium tape drives can we set /NOCRC and /GROUP=0

That depends on how much you trust the tape and the tape ECC and the media itself, and considerations siuch as whether or not you can fit your archival processing into your available window.

I know folks that do trust the tape ECC. And I know folks that don't. I tend to have some trust in DLT and SDLT and such, and DDS makes an exceptionally good doorstop.

[a version of this has been posted to the web site.]

Re: BACKUP uses nearly 100% CPU

Shriniketan Bhagwat — Wed, 18 Nov 2009 05:25:39 GMT

Hi,

What is the length of encryption key used?
Encryption strength is often described in
terms of the length of the encryption key
used to perform the encryption. In general,
longer encryption keys provide stronger
encryption and shorter encryption key
provides faster BACKUP.

Thanks and Regards,
Ketan

Re: BACKUP uses nearly 100% CPU

Volker Halle — Wed, 18 Nov 2009 07:02:48 GMT

Stephen,

if you are really interested (beyond just speculating), in which routines/libraries BACKUP spends it's CPU time, you can use the PCS SDA extension (PC-sampling):

$ ANAL/SYS
SDA> PCS ! to display help information
SDA> PCS LOAD
SDA> PCS START TRACE
... wait a while while backup is running...
SDA> PCS STOP TRACE
SDA> SET PROC/IND=
SDA> PCS SHOW TRACE/STATIS
SDA> PCS UNLOAD
SDA> EXIT

This will tell you, how often a specific PC value is being observed and SDA will try to symbolize those PC values. I assume you will recognize, where the majority of PC samples will be spent...

Note that a second CPU may NOT speed up this encypted backup, if the encryption operation cannot be multithreaded (is it using PTHREADS ?). A second CPU will provide capacity for other processes to execute.

Volker.

Re: BACKUP uses nearly 100% CPU

Nikus Giri — Thu, 19 Nov 2009 05:39:09 GMT

If you are using encryption, then it may consume high CPU. Why are you concerned with high rate of CPU consumption? Is it performance of the other application?
Currently, BACKUP does not have any control over data scan or throttling output.

Re: BACKUP uses nearly 100% CPU

Hoff — Thu, 19 Nov 2009 14:08:57 GMT

>Currently, BACKUP does not have any control over data scan or throttling output.

Output can be throttled with /IO_LOAD on OpenVMS versions where that qualifier is available (or if and as that gets back-ported?), and that tends to back-pressure the input scan.

the Alpha EV6 and EV7 processors don't compare well with Itanium nor all that well with current x86-64 processors; they're older.

If you need to offload the Alpha box, I'd tend to suggest using an Integrity as a BACKUP and compute server. The used hardware boxes and the current FOE base license can be quite affordable, though the cluster license cost was (when last I looked) comparatively huge (regardless of platform) part of any OpenVMS purchase.