SAM backup configuration

[Bill Hassell]

If you are using SAM for file backup, the default settings for tape have been obsolete for a long time.  The default values were setup for reel-to-reel tapes and small backup scopes. Here is the recommended settings to handle modern tape drives such as DDS (aka DAT), DLT, and LTO:

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

To change the default config for SAM, replace this file:
/etc/sam/br/fbackup_config
with the above lines.

This can improve the backup speed tremendously, especially for modern LTO drives.

If you run fbackup rather than using SAM, create the file: /etc/fbackup.conf
and copy the above text into this file.
Then always run fbackup adding the option: -c /etc/fbackup.config

[Duncan Edmonstone]

Bill, I was going to point out that fbackup/frecover have themselves "supposedly"* been deprecated, but then of course as you are talking about SAM, that is on 11iv1 (11.11) which itself is now out of support, so the point is probably moot! Anyway maybe worth adding that if you are on 11iv3 (11.31), you should use fbackup/frecover "at your own risk" as they are "supposedly"* deprecated from a support perspective.

*(I use the word "supposedly" here, because despite the man page stating deprecation for a long time now, HPE have still been releasing patches for fbackup/frecover for the last 10 years, the latest being in December 2020. This is HP-UX after all, it doesn't always make sense!)

[Bill Hassell]

Time for my annual rant about deprcating fbackup.

Simply put, without fbackup, HP-UX is sent back to the prehistoric age where tapes were reel-to-reel and backups amounted to a few hundred megabytes.

Without fbackup:

- Modern tape drives (LTO) cannot be kept streaming during writes
- Bundled tools for backup (cpio, tar, pax) have no index (the entire tape must be read for a list of files)
- Backups that exceed one tape are not possible or are very tricky to handle.
- There is no tape label so identifying when the tape was made is not possible without dreaded sticky labels.
- tar and pax can't backup files larger than 8GB, cpio can't backup files larger than 2 GB.
- No codes to support fast search marks that allow very fast recovery for selectedt files.
- No retries for locked files, locks are ignored and thus data can be corrupted.

(and other limitations unsuitable for an enterprise level computer system)

I have a feeling that deprecation of fbackup may have been due to the author retiring from HP and also the change to the fbackup format, making it incompatible with all previous versions (11.23 back to 10.20). The format was required to support uname up to 256 characters. To keep older versions from trying to deciphere the new format, an fbackup tape from an 11.31 system has the Magic Field:FBACKUP*LABEL whereas all earlier version back before 10.20 have the Magic Field: FBACKUP_LABEL. Note the * versus _ in the Magic Field. When reading an 11.31 tape on any previous version of HP-UX, you'll get the error:

frecover(5420): not an fbackup volume; magic value is incorrect

So backups from 11.31 can't be read on systems with HP-UX 11.23 or earlier. So the easiest solution was to label fbackup as deprecated, leaving 30 year old relics as the only backup solution that's part of HP-UX, (NOTE: frecover on 11.31 can read older tapes)

Well, it turns out that the 11.23 version of fbackup runs OK on 11.31. So for multiple OS versions, adding these files to an 11.31 system will keep fbackup tapes readable by earlier systems:

/usr/sbin/fbackup
/usr/sbin/frecover
/usr/sbin/fbackuprdr
/usr/sbin/fbackupwrtr

So why use (or continue to use) fbackup?

With a modern config file, LTO tapes will keep streaming during writes. This is done by launching 6 copies of fbackuprdr, each opening the next file and feeding the data into shared memroy area. This means that older systems with JBOD disks (common for internal drives) can keep feeding data faster than the tape needs. It shoiuld be noted that modern tape drives (LTO 4,5,6,7,8) are MUCH faster than spinning disks without an array controller. Recognizing this problem, LTO drives have the ability to slow down the write speed somewhat to accomodate slow disks. fbackup helps by keeping lots of file content in memory to keep the LTO busy.

FYI: streaming tape drives only have two modes: run, and stop. When a backup is running, if the disk data runs out (too slow), the tape drive stops, backs up to the last written record, then waits for more data. When the data starts arriving, the drive starts up, syncing with the previous data records and then adding the new data to the end of the last record. If this sounds tedious,it is for two reasons: tape drive performance will run vastly below normal speed, and the recording head and motors will be heavily loaded, leading to early failures.

Additionally, fbackup adds all the filenames to an index in memory. This is the first file on the tape and can be read in just a couple of seconds when first inserting the tape. Because fbackup supports multi tape backups, each tape always has the index recorded first, but with a marker that indicates where the new tape starts. So for a 5 tape backup, asking frecover to recover 1 file means that you put in the last tape, and run frecover normally. When frecover looks at the 5th tape's index, the marker indicates that it will be found on tape 3. Each tape has the tape number in the volume header. It also keeps track of the number of write sessions to the tape.

For performance, the config file can specify large records, also improving tape performance. The config file can specify the number of files to pass before recording a fast search mark. A low file count will generate a large number of markers, which themselves occupy extra overhead on the tape. But the markers mean that recovering a small number of files is much faster by skipping markers at a much faster tape speed.

There are no file size limits for fbackup so terabyte backups are no problem.

And there is very useful verify option (-N) which reads the entire backup looking for errors. By running this command:

frecover -rNv -f /dev/rmt/whatever > /tmp/frecover-scan.log 2>&1

the log file will show any files that were skipped because of locks. A file is often locked by databases because various parts are being changed. If the lock is ignored, fbackup could save portions of old and new data in the file, thus making the file incorrect when restored.

I will say that for larger systems, while fbackup is free, tape management (especially with autochangers) can become somewhat tedious, so a commercial backup application such as DataProtector is recommended over any of the included backup tools.

>> End of rant <<

[Duncan Edmonstone]

A wonderful rant Bill, and I won't disagree with a single word of it.

D

[Bill Hassell]

One other note:

Deprecation does not mean unsupported.

I recently submitted a defect report for frecover. The symptom was that a file that was actually on the tape could not be restored. In fact, the tape seemed to loop forever where this file was located.

It turns out (like many difficulties caused by backing up live systems) that if a file is significanlty reduced in size after the backup begins, the calculation to locate the file will overshoot the file and frecover will backspace to find the file...forever.

The fast restore feature using search marks will occasionally fail because if one or more files reducing in size during the backup. The actual space on tape no longer matched calculated space.

And it turns out that there has been an undocumented workaround for several years. If fast search marks is disabled, the file will be restored. This happens automatically when restoring everything since search marks are not needed. Same for network or fbackup file images.

But for tapes, the workaround was never documented. The solution is to use the -D option to disable fast searches for recovering selected files. The option has been hidden for several revs, but with the latest fbackup patch, the man page for frecover has been updated. Note that only a patch for 11.31 was created (PHCO_44859).  Here's the -D comment from the new man page:

      -D          This option disables index based fast search during the
                  selective retrieval of file from backup media. The file
                  retrieval using this option is much slower compared to
                  index based search and retrieval. This option should be
                  used to retrieve a file when frecover fails with an error
                  or enters an infinite loop attempting to find the location
                  on the backup media to retrieve the requested file.  See
                  WARNINGS for more information.

Thanks to the backline folks for digging this out and adding the documentation.

[Steven Schweda]

> - tar and pax can't backup files larger than 8GB, [...]

   Perhaps your old/obsolete "tar and pax" can't.  For many of us, such
limits disappeared decades ago.

      https://en.wikipedia.org/wiki/Tar_(computing)
      https://www.gnu.org/software/tar/manual/html_node/Formats.html

[Patrick Wallek]

Maybe better stated that the tar and cpio programs supplied with HP-UX have the above stated limitations.

Granted the GNU tar doesn't have those limitations, but I have dealt with some companies that do not allow non-commercial software to be installed on their systems, so they are stuck with the built-in tar, cpio, etc.

[Bill Hassell]

And to clarify what Patrick said, HP-UX hardware is generally purchased into large companies where corporate security prevents any non-vendor sanctioned software is allowedto be installed. The reference to SAM was intentional as there are hundreds of HP-UX sysadmins that have little or no experience except the Google click-school. (tm). My comments about pax and tar refer to the HPE bundled and approved versions.

Even those sysadmins that can download freeware versions of tar or pax, all the issues about performance and tape management still apply.

[Duncan Edmonstone]

I thought pax and specifically PAX-Enh package removed these limitations and support both tar and cpio formats:

https://myenterpriselicense.hpe.com/cwp-ui/free-software/PAX-Enh

D

[Bill Hassell]

Yep, I saw that sometime ago. But since I'm working with systems from 10.20 on, and many never patch their systems, the Enh-PAX is more of a footnote. And it does require redaing the man page and adding the -x pax option.

But even with large file limits removed, it is still a single threaded app that will wear out tape drives with stop-backup-restart (shoeshining) pauses because the data isn't flowing fast enough. And all the high speed recovery, index file info, volume headers, etc are still absent.

And in case someone doesn't know, the LTO-3 drives need 160 MB/sec to keep data flowing in compressed mode. Jump to LTO-8 and the compressed transfer rate jumps to 1180 MB/sec, way beyond any spinning disk capability, and even pushing smart array controllers managing lots of striped disks. SSD can approach this speed but for very large backups (terabytes), most of the storage won't be SSD. That's why fbackup can use 6 reader processes to stay ahead of the tape drive.