HP-UX Backup failed from last two days.

[Prashantj]

Hi,
We have hp-ux 11.11 with 8 slot tape library.
We have facing backup failed issue from last two days. On Sunday we have manually run the script bcoz script not executed as per crontab.
Backup script stop after step#3 , but till Monday filesystem mirror ( /m in front of filesystem ) hence I run step#4 to unmount mirror disk.
I have checked all logs but didnâ t found any related / error logs in system.
Please find the attached script and his logs entries. In Script after step #3 logs not generated.

Appreciate if you could provide the solution or reason.

Thanks in Advance.

Prashant

[johnsonpk]

Hi Prashant,

It seems your step#3 is indefinitely hanging ( backup session never exits)

Try executing your step 3 manually and see
/usr/sam/lbin/br_backup DLT FULL Y /dev/rmt/c7t1d0BEST /etc/sam/br/graphfullm root Y 1 N

In your main script the step4 is getting executed only after step3 and the step3 script/command never returning to its parent shell.

it may be because of a bad tape or problem with the drive


Rgds
Johnson

[Steven E. Protter · Accepted solution]

Shalom,

Note: Your backup methodology is slick, breaking the mirrors to get clean copies of everything. But there are problems.

1) You can not restore a system with this backup. If you lose the boot disk, you need an Ignite make_tape_recovery or make_net_recovery to have a way of recovering the system itself without a cold OS re-install

2) Breaking mirrors on running databases does not insure you will get clean copies of hot databases. A checkpoint must be written to insure your oracle backup is not fuzzy.

The most common cause of fbackup failure is running out of space on the tape and needing to manually change media and tell the backup to continue.

There may be an issue with breaking one of the mirrors. Possibly stale extents. lvdisplay -v the logical volume and pvdisplay -v the physical volume involved.

SEP

[TTr]

I know you said you checked the logs but did you specifically check the /var/sam/log/br_log?

Most likely the tape got full. In your log, the backup process read 117,373,401 blocks and wrote 117,899,314 (that may be accounting for the inter-record marks as well) but overall assuming 1024-byte blocks you have writen about 117GB of data to a DLT tape. That's about 3-to-1 compression ratio which justifies a full tape condition.

You should consider replacing the sam br_backup utility with the real fbackup utility and utilize the chvol feature of fbackup.

Also, I did not see any interaction with the database before you split the mirrors. With this setup, at best, if you need to restore the database, after restoring your database files, you will have to go through a crash recovery procedure of the database. You should consider putting the database in hot-backup mode (archive log mode) right before you split the mirrors and put in in regular mode as soon as the mirror splitting is done. No need to keep it in hot-backup mode while the backups are running.
I assume the archive logs are also backed-up by this or another job at about the same time.

[Prashantj]

Hi John,Steven & TTr,

Thanks to all for your quick reply and valuable information.
< /var/sam/log/br_log ? >

Logs not available for Sunday.
Also I checked /var/adm/cron/log details as below

! *** cron started *** pid = 5399 Sun May 31 02:35:31 EST 2009
> CMD: /usr/adm/cron.jobs/hm.rps.cost.job #xxx Cost Update XXXXXX
> root 5322 c Mon Jun 1 05:40:00 EST 2009
> CMD: /var/adm/cron.jobs/edifilterin_ky.sh >> /var/adm/cron.jobs/edifilterin_ky.out 2>&1
> root 5323 c Mon Jun 1 05:40:00 EST 2009

For the Monthly activity we reboot the production server. In that log cron started at 02:35 EST 31st May and after that next entry on 05:40 EST 1st June, in between cron logs not generated that means cron daemon
Not successfully started & for this reason backup script not executed.

[Sunny123_1]

Hi Prashant

Your cron daemon not started properly.That's why your backup got failed.Now your daemon is working fine.Please check your next backup logs.

Regards
Sunny