Oracle control file corrupted when the backup were running

[Hugo Melo]

Hello All, I'm DP admin in my job. Last weekend while an incremental oracle backup were running, specially to remove archive files, the Oracle control file got corrupted. Our DBA team were indicating that the cause of problem were the backup process.
So I'd like to verify with all of you guys, if anybody has heard anything about DP affecting the oracle environment. Any comment or suggestions will be apreciated.

[TTr]

In HP-UX you can read a file with the intent to back it up without having to lock it or even if it is open by another app. In windows you can not do this and maybe that's what your DBA is thinking of.
Did this happen once or it happens everytime you run backups? Or did it happen as soon as you started backing up with DP?

You also mentioned archive logs, normally you don't backup the control files (or the redo logs or the data files)? What exactly did your backup job do?

[Horia Chirculescu]

Hello Hugo,

Maybe you could share with us what kind of backup are you performing? Do you use RMAN ? Do you use some custom scripting to perform backup?

You say that you are DP admin. Is this a typo? If not, can you please explain to me what does it mean?

I suppose your ORACLE DB have 2 or 3 control
files, right? All of those files were corrupted? Do you have a full backup or not?

It is a good idea to actually keep in handy (and to make it on regular basis) a full backup.

Best regards,
Horia.

[Hugo Melo]

Thanks all of you to your prompt response! Just one clarification when I said that I'm Data Protector admin that mean I'm the person which is in charge of DP.
The backup job is using the RAM, from oracle side without any different scripts.
From oracle stand point I can comment to you that we have 3 control files. When the problem happens just one got corrupted.
Our backup schedule is distributed as indicated below, and it is executed in a daily basis:
0:00 Full Backup of Oracle
06:00am / 01:00pm /09:00pm Incremental backups with focus on archive files removal.

I faced this problem last weekend and it has happened exactly 30 day before. I mean happened 2nd weekend of december/2009 and now of january/2010.

[TTr]

> I'm the person which is in charge of DP.
Is there a UNIX admin? Did he/she look at this issue?

> The backup job is using the RAM, from oracle side without any different scripts.

???

> 0:00 Full Backup of Oracle
> 06:00am / 01:00pm /09:00pm Incremental backups with focus on archive files removal.

Which files are in the datalist of the DP backup jobs? If RMAN is running, then RMAN is backing up the oracle databases and usually RMAN is set up by the DBAs. Then you (the backup admin) only need to backup the RMAN areas and NOT the database files. So the DP most likely is not touching the control files unless your datalist is including them. You need to provide more details here.

[Rita C Workman]

Put simply - Data Protector just backs up files that are not locked. Period. So if your Oracle is in a proper hotbackup mode, you can back it up.
If the file is corrupted on the disk - then Data Protector will back up a corrupted file.

One thing...you noticed a pattern that started last month. You might want to check and see if the DBA's are running a once a month process that would affect Oracle not running RMAN.

If your not the HPUX Admin, then ask them to check cron for Oracle. Or check with DBA's and maybe they can confirm. Even if they don't and you mention this - they might go back and change when they are running this and then if the problem 'quietly' goes away - you'll know!

Rgrds,
Rita

[Hugo Melo]

The guys from UNIX, actually HP-UX 11.11, also were involved in this case. They crosschecked the entire server but they found no problem or possible cause. From their perspective the OS is working fine and believe isnâ t the problem cause.

Iâ d like to clarify again that Iâ m backing up the Oracle using the RAM. So for those Cell Manager which has Oracle integration, you can setup backup jobs for oracle without using â externalsâ RMAN backup scripts.

[Horia Chirculescu]

As Rita said "If the file is corrupted on the disk - then Data Protector will back up a corrupted file.".

I believe Oracle recommend that each control file to be on different physical disks. If this is the case, maybe the HP-UX admin should check for hardware problems on the physical disk which holds the corrupted file.

Another thing: I am not very good at acronysms, can you please explain what it is RAM?

I understand that your daily backup procedure is

> 0:00 Full Backup of Oracle
> 06:00am / 01:00pm /09:00pm Incremental >backups with focus on archive files removal.

All three incremental backups experience the same corrupted control file? Your full backup is all right, no corruption?


Best regards,
Horia.

[Eric Antunes]

Hi Hugo,

First of all, DP has nothing to do with this controlfile corruption since all DP does is an interface with RMAN (not RAM). So, the issue is with RMAN (DBA's job) or filesystems (HP-UX adm job's).

Best Regards,

Eric Antunes

[John Brooks_1]

Hugo,
RMAN stores backup information in the controlfile and will increase the size of the controlfile, make sure it has room to grow. You can use the Oracle parameter CONTROL_FILE_RECORD_KEEP_TIME to control the number of days to retain the information. Mine is set for 90, but should reflect your tape retention period.

[Hugo Melo]

Thank you John. We reviewed the filesystems (3 different on our environment) that is used by the control files; it has 80% of usage. When the problem happened just one got at 100% of usage. The others (2) kept in the same size.
Anyway, you comment was a good one, I will be forwarding this indication to our DBAs.

[TTr]

> When the problem happened just one got at 100% of usage.

That's the problem right there. When the filesystem is full, any data writes to a file will fail and leave the file corrupt. Your UNIX admin should have caught this and give this as the reason for the corrupt control files.

[Hugo Melo]

Thank all of you for your atention and comments on my case.

[WayneHP]

One point. If you only corupt one(Or even 2) of the three control files, you can copy any "good" control file over the courpt control file with the DB DOWN and restart.

As long as you have enough disk space.