Problem copying file

[Olga_1]

Hello,

I have HP-UX B.11.23 U ia64 server.
There is an intermittent problem to copy file from one FS to another using script.

Manual copy works OK.

Script looks like this and suppose to copy 74 database files:
cp app_data06.dbf /u04/oradata/MSSPROD/app_data06.dbf
cp mssprod_app_data01.dbf /u04/oradata/MSSPROD/mssprod_app_data01.dbf
.... etc

At one point it would exit with Permission denied error. (Permissions and space are not an issue.)
If I check the destination it has already some files copied and one file that looks like half copied:

-rw-r--r-- 1 oracle dba 1572872192 May 25 11:02 mssprod_app_data10.dbf
-rw-r--r-- 1 oracle dba 2097160192 May 25 11:03 mssprod_app_data11.dbf
-rw-r--r-- 1 oracle dba 2097160192 May 25 11:07 mssprod_app_data12.dbf
---------- 1 oracle dba 4718600192 May 25 11:00 mssprod_app_data13.dbf


Any ideas?
Thank you,
Olga

[Andrew Young_2]

Hi Olga

My guess is that the filesystem you are copying to is NFS mounted. There is a filesize limit of 4GB for an NFS mounted filesystem.

The filesystem copy has stopped but because of the error the permissions have not been set yet.

Regards

Andrew Y

[Olga_1]

Thank you for the reply, Andrew but I do not copy to NFS. And as I mentioned, if I re-ran the script it works. The problem is intermittent.

[Geoff Wild]

Change script to remove destination files first:

rm /u04/oradata/MSSPROD/*.dbf

then run the copy.

Rgds...Geoff

[Olga_1]

Geoff, I cleaned up the files before running the script. The destination is empty.

[Geoff Wild]

Okay - need more input please:

bdf /u04/oradata/MSSPROD/

bdf /source

(where source is the source directory)

Is oracle id doing the copying? (I'm assuming as you are not doing a cp -p)

ls -ld /u04/oradata/MSSPROD/

ls -ld /source

Is the db down when you copy the files? (it better be - for you can have issues trying to copy files that are busy).


Rgds...Geoff

[Andrew Young_2]

Hi Olga

OK. Sorry about that... my bad - its 4TB limit for NFS 4 in any case.

Some question that might point us in the right direction.

Are there any core dumps?

What version of JFS are you running? How recently was it patched?

Is there anything special about the disk or device that you are writing to?

Are there any entries in the /var/adm/syslog/syslog.log?

What user is this running as?
What shell is this running in?
What is the result of the ulimit -a command?

HTH

Andrew Y

[Olga_1]

Geoff, here it is and thank you for your help.

Destination:
Filesystem kbytes used avail %used Mounted on
/dev/vg03/u04 35848192 25869 33583435 0% /u04

Source
Filesystem kbytes used avail %used Mounted on
/dev/vg03/u77 166920192 156575189 9698470 94% /u77



Is oracle id doing the copying? Yes

ls -ld /u04/oradata/MSSPROD/
drwxr-xr-x 2 oracle dba 1024 May 28 11:08 /u04/oradata/MSSPROD/

ls -ld /source
drwxr-xr-x 2 oracle dba 4096 May 25 13:02 /u77/staging

Is the db down when you copy the files? DB is down

[john korterman]

Hi Olga,

your destination directory was last changed around the time when the file causing the "permission denied" error could have been finished. Are you absolutely sure that noone has tampered with the destination directory permissions?

regards,
John K.

[Geoff Wild]

Only thing that stands out - is the size of /u04 - 33 GB - what is the size of all the db files?

Looks like source is approx 160 GB...

Rgds...Geoff

[Olga_1]

Andrew, here what you asked for:

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4194300
stack(kbytes) 131072
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

OnlineJFS01 4.1.008

Disks: EVA

User: oracle

SHELL: ksh

No errors in syslog.log?

John, I am sure that nobody changed the permissions.

Geoff,

Files take about 31G, so space should not be an issue.

Thank you all.

[john korterman]

Hi again Olga,

the last modification date of the file mssprod_app_data13.dbf in the destination directory seems to be before the files whose sequence number woukd date them earlier. Are you sure that mssprod_app_data13.dbf in the destination directory does not belong to a failed earlier transfer, and you cannot overwrite it because you do not have the right to do so?


regards,
John K.

[Olga_1]

Sorry John for the confusion, but in my initial post it is not the real look of the directory when the problem happened. Since it is hard to reproduce it I simulated how directory looked after the script failed. So do not pay attention on time stamps.

[Jov]

Hi Olga,

Have you done a full trace of the script and see what and when it fails on? Also was it consistent with a particular Ora datafile?

set -x the start of the script and log the output to /tmp and see if you can profile the problem?


Jov

[john korterman]

Hi again Olga,

are the permissions displayed correctly? I assume that you will get a "permission denied" error if you as user=oracle tries to overwrite mssprod_app_data13.dbf as is.
A file without permissions is normal in situations when the destination file system ran full during copy. Although you have stated that space is not an issue, please check the syslog for a possible message of af full destination file system.

regards,
John K.

[Andrew Young_2]

Hi Olga.

Please run ulimit -H -f and ulimit -S -f as the oracle user. This will give the and hard soft limits in terms of bytes being written. The figure given is in 512K blocks.

I'm beginning to think your permission denied error may be caused a user limit or quota condition kicking in at a shell or kernel level preventing you from writing that data.

Because it is a hard/soft limit it is only kicking in intermittently When you rerun the script it ignores the data already copied perhaps.

To change these values to unlimited use the following commands:

ulimit -H -f unlimited
ulimit -S -f unlimited

Note these unlimited settings will affect some of the oracle initialisation scripts since they do not like the unlimited option (unique to the HP ksh)

HTH

Andrew Y

[Olga_1]

Andrew, thank you very much for not giving up.

Both settings are set to unlimited:

ulimit -H -f
unlimited
ulimit -S -f
unlimited

[Andrew Young_2]

Hi Olga

That was also not correctly read by me. The file write is already set to unlimited, ignore that posting.

However maybe an increase in the data blocks might make a difference.

Maybe run sar -v 2 120 during the copy to see the effect during the copy. May also use Glance Plus to monitor system activity and utilization during the copy to see if any resources are getting stretched thin at the time

Regards

Andrew Y

[Dennis Handly]

You may want to change your cp to cp -f. That way any leftover files with no permissions won't stop things.

Though you did say you removed the files first.

[Jov]

Is it normal behaviour that destination file gets its permission set after the completion of the copy?

I thought it was dependent on the user and directory settings.

How often does this problem occur, or has it ever completed properly?


Jov

[Dennis Handly]

>Jov: Is it normal behaviour that destination file gets its permission set after the completion of the copy?

I believe so, yes. It is a feature such that the file can't be fiddled with, until after it is copied.

[Jov]

>> I believe so, yes. It is a feature such that the file can't be fiddled with, until after it is copied.

Its been awhile since I last check a large file copy and thought this was the case, but not sure if it was Solaris or HP-UX.

The only other think I can think of that could cause this issue is Oracle doing some kind of lock on the dbf datafile during access. Especially give the amount of time it takes for these datafiles to copy, Oracle accessing them would be pretty common on a moderately busy database instance.

Can I ask why are you using a script to copy the datafiles and not one of Oracle's backup technologies?

Jov

[Geoff Wild]

The other thing I thought of - why not stream multiple cp's at once?

Or maybe use fbackup | frestore?

fbackup -i /u77/staging
-f - | (cd /u04/oradata/MSSPROD; frecover -Xrf -)

Rgds...Geoff

[Rasheed Tamton]

Hello,

Can you test the script by running as root instead or oracle and see whether you get the same error.

-Are you using full path when copying from and to
-How about giving write access to the group owner (dba) and see whether you get any issue with it.

Regards,
Rasheed Tamton.