frecover takes very long time

[kenken_3]

Hi,

We need to help to restore some oracle data for our customer. They use fbackup and LTO2 for the backup and the OS is 11.23 Itanium. I use LTO4 ESL tape library to restore the tape.

In the fbackup, there are many small files and a few very large files. When I try to restore the single large file(60G), it takes me almost 2 days (the restoration speed is like 1.5G/hour). I am afrad it is too slow.

by the way, when we try to recover the file, we always get the message:
frecover(2105): did not find expected file marker
frecover(5419): checksum on volume header is incorrect.
frecover(5420): not an fbackup volume; magic value is incorrect
frecover(2105): did not find expected file marker
frecover(5412): Do you wish to try to salvage this volume? (^[yY]/^[nN])

but I am still able to proceed.

Anyone encountered the same issue? please share it.

thanks

[kenken_3]

I have very high memory utilization and IO usage.

GlancePlus C.04.50.00 14:28:27 phs71626 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU | 4% 4% 31%
Disk Util F F | 83% 81% 100%
Mem Util S SU UB B | 90% 90% 90%
Swap Util U UR R | 23% 23% 23%
------------------------------------------------------------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 2
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 532 39608 110.8 128.9 2283.8
Page In 0 19 0.0 0.0 730.3
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 76kb 0.0 0.2 18.8
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 42 0.0 0.1 2.9
VM Writes 0 350 0.0 1.1 29.1

Total VM : 799mb Sys Mem : 1.1gb User Mem: 503mb Phys Mem: 4.0gb
Active VM: 657mb Buf Cache: 2.0gb Free Mem: 406mb


any kernel parameter can be tuned for this?

[Duncan Edmonstone]

what OS version was the fbackup taken on? 11.23 as well, or did it come from a 11.31 system?


Only reason I ask is this message in the man page on 11.31:

In HP-UX 11i Version 3, the maximum value for fields returned via uname() was increased (from 8 to 256). To accommodate the larger size, a format change was necessary. A new magic number, FBACKUP*LABEL, was created to distinguish this new format.

Likewise with HP-UX 10.20, HP-UX added support for large files (greater than 2GB) and increased UID/GIDs (greater than 60,000). The magic number associated with this release through HP-UX 11i Version 2 (inclusive) is FBACKUP_LABEL.

Archives and files with formats and attributes that are unsupported on previous HP-UX releases could cause severe problems or unpredictable behavior if attempts were made to restore onto these systems. For this reason, fbackup creates tapes with a magic number that is only recognized on releases which support the features and format being archived. This prevents fbackup tape archives from being restored on earlier HP-UX systems than are supported. frecover still reads all tape formats so that fbackup tape archives created on earlier HP-UX systems can be restored.


Which is here:

http://docs.hp.com/en/B2355-60130/fbackup.1M.html

Suggests the sort of errors you see.

You don't have high IO or memory issues - that's not your problem.

HTH

Duncan

[kenken_3]

the backup was taken from HPUX11.23 as well. I have very high disk IO now.

Disk Util F F | 83% 81% 100%
Mem Util S SU UB B | 90% 90% 90%

[Duncan Edmonstone]

No, you don't have very high disk IO.

All that is telling you is that the busisest disk in your system is doing some sort of IO operation for 83% of the time during the emasurement interval. A disk doing some kind of IO operation for 83% of the time does *not* indicate high lelevls of disk IO.

Anyway, back to your problem... it looks to me like for some reason the backup isn't good... I'm guessing that in your case fbackup is no longer looking for file search markers which allow it to fast serach through to the file it wants for restore - so things take a long time as it has to look at every file as it spools through the tape.

If you aren't in a position to try another backup, or take a new backup, I'm not sure there's an easy answer to your problem.


> frecover(5420): not an fbackup volume; magic value is incorrect

I guess you could take a look at what header *is* on the tape:

frecover -V /tmp/fbup.hdr -f

obviously replacing with /dev/rmt/0m or whatever is appropriate for your system...

This should pop a copy of the header into the specified file for you to look at - maybe you'll get a clue as to what the problem is in there...

HTH

Duncan

[kenken_3]

Thnaks. I will give it a try.


does the /dev/rmt/1m or /dev/rmt/1mn matter?

they can use /dev/rmt/1m at the source server, but I only can use /dev/rmt/1mn in the new server.

[Duncan Edmonstone · Accepted solution]

>> they can use /dev/rmt/1m at the source server, but I only can use /dev/rmt/1mn in the new server.

ahh this might be related to your problem.

I'm guessing the new server has SAN attached tape drives yes?

If that's the case, then many people will block access to all rewind-on-close tape devices using the kernel parameter st_san_safe.

That's fine for most backup software - but fbackup specifically wants to use the rewind-on-close tape device - in fact if you read the man pages it tells you it *must* use the rewind-on-close devices...

(yes this _does_ mean you can't really use fbackup on systems with SAN attached tape drives, but hey, the fbackup tools are bing deprectaed now wanyway so you should be looking at alternate backup tools anyway)

So temporarily at least I would try setting the st_san_safe kernel parameter back to 0, and then try the restore again with the rewind-on-close device - i.e. /dev/rmt/1m

kctune st_san_safe=0

should do the trick - can't remember if it's instant or needs a reboot though...

HTH

Duncan

[kenken_3]

Duncan,

after I change it, the recovery speed is much faster now.

thanks.