Exit Code of 2 with my LTO3 tapes

[dx12180]

Hello world, I am very new to the HPUX world. Trying to do automated backups at night with the SAM backup. My DB server is 263 GB. I am using LTO3 tapes which should be suffice to back it up but I still get an error code of 2.

This is my maching info.

CPU info:   1 Intel(R) Itanium 2 9000 series processor (1.4 GHz, 12 MB)           532 MT/s bus, CPU version C2           2 logical processors (2 per socket)

Memory: 20449 MB (19.97 GB)

Firmware info:    Firmware revision:  02.03    FP SWA driver revision: 1.18    IPMI is supported on this system.    BMC firmware revision: 5.14

Platform info:    Model:                  "ia64 hp server rx3600"   

  Release:   HP-UX B.11.31   

These are the scripts I used at night for backups.

15 20 * * 2,4 /usr/sam/lbin/br_backup LTO FULL Y /dev/rmt/4mn /var/sam/graphZBAa13454 root Y 1 N > /var/sam/SAM_br_msgs 2>&1 #sambackupdb

15 20 * * 1,3,5 /usr/sam/lbin/br_backup LTO FULL Y /dev/rmt/4mn /var/sam/graphZBAa07658 root Y 1 N > /var/sam/SAM_br_msgs 2>&1 #sambackupdb

I have changed out the cable, tape drive, and fed it with new tapes from HP. I have checked all three logs, br_index.full, br_log, samlog but they only tell me that I am a failure at backing up.

I dont really know what else to do, please help. 

[Patrick Wallek]

What does this log show -- /var/sam/SAM_br_msgs

This should contain most of the messages from the backup.  Have a look at this log, or attach it so we can look at it.

[dx12180]

fbackup(1004): session begins on Thu Aug 15 08:47:01 2013
fbackup(3205): WARNING: unable to read a volume header
fbackup(3024): writing volume 1 to the output file /dev/rmt/4mn
fbackup(3009): WARNING: File number 10646 (/oracle/app/diag/tnslsnr/MNPFODB/list
ener/trace/listener.log)
        was not successfully backed up
fbackup(3009): WARNING: File number 95422 (/oradata/MN/comp_1.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95423 (/oradata/MN/compindex.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95432 (/oradata/MN/sidpers_data.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95434 (/oradata/MN/sysaux01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95435 (/oradata/MN/system01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95442 (/oradata/MN/undotbs01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 95451 (/oradata2/MN/redo4_b.log)
        was not successfully backed up

[Matti_Kurkela]

Are you trying to backup an Oracle database with fbackup while the database is active and in use?

The "File number ... was not successfully backed up" messages would indicate that the data in the database changed while the backup was running. That would mean the backup is not consistent: some parts of the backed-up database are from before the change, and others from after the change. An inconsistent file-based backup of a database can range from "slightly corrupted, fixable with some extra work" to "completely useless waste of time and effort".

Backing up a database with purely file-based backup tools would require the database to be stopped (or placed in a special backup mode) while it's being backed up. If you cannot afford the downtime to do that, you'll need an on-line backup solution for your database.

This is by no means specific to HP-UX: it is more like "Database Backups 101".

[dx12180]

Sorry about the confusion. While I was writing this I was running a backup for testing purposes. I used a different graph file which included the oracle and oradata dir which I know is constantly changing. I did get an exit code of 4 even though those files did not backup.

Tonight I am going to back up the other graph file which I have been getting the exit code 2, here is the directories of the graph file.

DB@/root:ROOT->more /var/sam/graphZBAa07658
i /
i /oracle
i /usr
i /var
i /tmp
i /opt
i /locp
i /sidpers
i /stand
i /archive
i /sidpers_hist
i /sidpers/ups
i /sidpers/sysadm
i /sidpers/reports

 We have 2 online backup that we used as well

[dx12180]

This is what I get when running the above graph file:

br_backup: Invoking fbackup. See /var/sam/log/br_log for details.
fbackup(1004): session begins on Thu Aug 22 08:05:02 2013
fbackup(1517): /comp not backed up - 'n' option (NFS) not specified
fbackup(3203): volume 1 has been used 1 time(s)
fbackup(3024): writing volume 1 to the output file /dev/rmt/4mn
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99257_614783317.arc.gz
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99258_614783317.arc.gz
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99259_614783317.arc.gz
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99260_614783317.arc.gz
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99261_614783317.arc.gz
fbackup(1102): WARNING: unable to stat file /locp/hotb/archive/MN/1_99262_614783317.arc.gz
fbackup(3005): WARNING: file number 16035 was NOT backed up
fbackup(3005): WARNING: file number 16036 was NOT backed up
fbackup(3005): WARNING: file number 16037 was NOT backed up
fbackup(3005): WARNING: file number 16038 was NOT backed up
fbackup(3005): WARNING: file number 16039 was NOT backed up
fbackup(3005): WARNING: file number 16040 was NOT backed up
fbackup(3009): WARNING: File number 99325 (/oracle/app/diag/tnslsnr/MNPFODB/listener/trace/listener.l
og)
        was not successfully backed up
fbackup(3009): WARNING: File number 184052 (/oradata/MN/DBA.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184054 (/oradata/MN/FLIPL.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184064 (/oradata/MN/comp_1.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184065 (/oradata/MN/compindex.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184074 (/oradata/MN/sidpers_data.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184076 (/oradata/MN/sysaux01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184077 (/oradata/MN/system01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184082 (/oradata/MN/temp_users.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184084 (/oradata/MN/undotbs01.dbf)
        was not successfully backed up
fbackup(3009): WARNING: File number 184094 (/oradata2/MN/redo5_b.log)
        was not successfully backed up
fbackup(3003): normal EOT
fbackup(3310): enter '^[yY]' when volume 2 is ready on /dev/rmt/4mn,
 or '^[nN]' to discontinue:
fbackup(3004): writer aborting
fbackup(1002): Backup did not complete : Reader or Writer process exit

[dx12180]

I believe the data might be too large and is asking for a second tape. But I am running on 400/800 LTO3 tapes. I calculated all the directories in the graph file and its around 275.4 GB. Not sure what the issue is. Please help.

[Dennis Handly]

>I believe the data might be too large and is asking for a second tape.

Are all those "not successfully backed up" causing multiple copies written to the tape?

Also, why not provide another tape to see how many it will take?

[dx12180]

The SAM_br_backup is automated so it just times out and abort the process. I will try the regular fbackup command next to see if it ask for another tape and let you know.

[Bill Hassell]

>> fbackup(3003): normal EOT

Regardless of what you calculated, the amount of data has exceeded the length of the tape. So the message is normal. You may want to use du -sk to summarize the amount of data. Also, a dirty tape of drive can cause excessive retries and long erase-forward sequences to skip the bad spots. 

Run a cleaning tape followed by a new tape.

[dx12180]

Today, I got an exit code of 4. Couple of things I changed.

1. I changed where the log file location within the script

2. Changed the blocksperrocord from 32 to 128 in fbackup config file

3. Changed the checkpointfreq from 32 to 256 in fbackup config file

Here is the results from my log file: Not sure why the last line was cut off after "WARNI"

br_backup: Invoking fbackup. See /var/sam/log/br_log for details.
fbackup(1004): session begins on Thu Aug 22 20:15:01 2013
fbackup(1517): /comp not backed up - 'n' option (NFS) not specified
fbackup(3203): volume 1 has been used 2 time(s)
fbackup(3024): writing volume 1 to the output file /dev/rmt/4mn
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/2/1019/Failu
res.cvf
fbackup(3005): WARNING: file number 85876 was NOT backed up
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/2/1021/Failu
res.cvf
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/CV_JobResult
s/2/0/753727
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/CV_JobResult
s/2/0/753727/State
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/CV_JobResult
s/2/0/753727/backup.out
fbackup(1102): WARNING: unable to stat file /opt/simpana/iDataAgent/jobResults/CV_JobResult
s/2/0/753727/raw.log
fbackup(3005): WARNI

[Bill Hassell]

>> Changed the blocksperrocord ...

fbackup has a very archaic (read: reel-to-reel) default for records. Not only is it inefficient, but it will be quite slow.

Replace your config file with these values:

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

 This should improve your performance significantly.