frecover hangs or fails to restore selected files

[Bill Hassell]

frecover warning

fbackup and frecover have been the best tools for backup and recovery for some 30+ years of HP-UX. The pair appeared at the first release of HP-UX 10.00 during the early 1990's. It is still used on the majority of smaller HP-UX systems, partly due to excellent performance and also the ability to handle the multi-GB files. A few years ago, fbackup was deprecated by HP in favor of pax for backups. But as many HP-UX admins discovered, there are several issues with pax.

First, fbackup is an HP-proprietary tool and format. It cannot read or write tar, pax, cpio, etc formats. So to interchange files between other Unix-style systems, the choice must be tar or pax. But all of the classic Unix-style tools have very poor performance for tape recovery. These tools are serial-only, that is, they must read the entire tape archive to look at a table of contents. And they encourage poor sysadmin practices such as appending multiple backups onto the same tape.

In troubleshooting an frecover issue, I uncovered an old bug that hasn't been well documented. While restoring complete backups, recovering specific files or directories (using the -x option) can hang with apparently no tape activity or show an error message about being unable to recover. The good news is that there is a workaround.

First, the problem:

frecover uses a feature of most streaming tape drives called fast search marks. DDS and LTO drives can change from serial reading to a high speed skip between search marks. This can reduce the recovery time for a file at the end of the tape to just a few minutes. And in order to determine how many search marks to skip, the INDEX file at the beginning of the tape is consulted. However, the INDEX file is created before any files are stored. This INDEX is all the files specified for the backup. It is written as the first file on the tape. That is why the INDEX (frecover -I -) takes only seconds to read.

However, all backups take time, and as a result, things may have changed during the backup period. Files might get created (and therefore not backed up), other files might be deleted (frecover generates a message during backup) or files may change size. And that is the problem area. If a file is reduced in size, the search mark positioning (based on the INDEX) may pass over the actual file location and frecover will try endlessly to reposition the tape to the previous mark and try to find the file or simply give up with an error message. For the hang condition, here are no error messages or logs for this condition.

Note that this isn't a common problem. Several conditions must exist at the same time: changing file sizes during backup, and restoring a specific file that follows changed files. This hang was documented at HP in 2001 (KBRC00003769 with SR 8606155704) but was not communicated very well. A couple of additional service requests (emr_na-c00930187 from 2008 and emr_na-c02635744 from 2010) were also submitted to HP.

The solution is to disable fast search mark positioning by defining the tape devicefile as a network device, something like this:
# frecover -f myhostame:/dev/rmt/0mn ...

myhostname would be the string from the hostname command.
However, to get this to work, the remsh command must be enabled (/etc/inetd.conf) and root's $HOME will need a .rhosts file with the hostname followed by the word root. Test this with the command:

# remsh $(hostname) date

It should return the current date without needing a login.
But remsh (indeed, all the 'r' commands like rlogin, rcp, etc) is considered to be a network security risk and is deprecated by most network administrators.

However, undocumented from the first version of frecover, is the runtime option -D. Both the -D as well as the myhostname:/dev/rmt feature accomplish the same thing by not using fast search marks. In this case, the tape is read serially, looking for the desired file or directory. In this case, the INDEX is only used to determine if the requested file is actually on the tape before starting to read.

HPE has recognized the oversight in not documenting the -D option and has issued a Customer Advisory. For those of you are not on the distribution list, here is the advisory:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00097577en_us

The -D option has been present in all versions of frecover so it can be used as needed.
Note that -D turns off the fast search capability, so selective file recovery will be much slower. Use -D if you experience a hang or error message while restoring.

Also not that early in the 11.31 release cyclce, fbackup/frecover were enhanced to accomodate a slightly used feature that required changing the format of the archive. While the latest frecover version can read all versions of fbackup archives, frecover versions prior to 11.31 cannot read 11.31 fbackup tapes. This incompatibility was not popular and may have been the underlying reason that fbackup was deprecated in 11.31 documentation. The good news is that earlier versions of fbackup can be copied to 11.31 systems and used to create backward compatible tapes. You would need both the frecover and fbackuprdr executables to create archives from 11.31 to be used in earlier HP-UX versions such as 11.23 or 11.11.

As far as deprecating fbackup in favor of pax:

When this was announced (several years ago), the general feeling from customers at many user group meetings was that pax was a very poor replacement for fbackup.

Here are the failings for pax:

No INDEX file at the front of the tape. This means that a table of contents requires serially reading every file on the tape, perhaps an hour or two for large backups just to see the file list.

frecover uses fast search marks that are available in most tape drives for the last 20 years. pax must serially read the entire tape even if the needed file is located early. The reason is that pax (like tar) allows appending the same files to a single tape. So to get the latest file, the entire tape must be read.

No high speed match for selected files or directories. This means that a spelling error for a required file requires reading the entire tape before reporting that the file was not found. frecover reports in a few seconds when a selected file name is not on the current tape by using the INDEX at the beginning of the tape.

pax (like cpio and other archaic backup tools) allows appending to the tape. This means that the same file could appear on the same tape. Recovering a specific version of the same file requires very careful records. fbackup will not append to any tape as it always rewinds prior to starting the backup. The INDEX file always reflects what is on the current tape.

pax cannot backup any file larger than 8 GB. 20 years ago, this was a reasonable limitation and followed industry standards such as cpio and ustar. fbackup has no documented file size limitation (terabytes are fine).

pax has *NO* multi-tape backup capability. When the tape is full, the pax is aborted. On the other hand, fbackup identifies each new backup volume and marks each tape's INDEX file with volume identifier. The first tape will have everything marked as the first tape, then subsequent INDEX files on each new tape will update the location of each file. The last tape in a multi-tape backup documents all the volumes. So by inserting the last tape and recovering a file, frecover will report the actual tape needed for the recovery.

These issues make pax a very poor replacement for fbackup. I'm aware of the recommended backup tool: Data Protector, but for the majority of small to medium data centers, price, backward compatibility issues and the sheer complexity in running Data Protector has made it a poor replacement for fbackup.