topic Re: Problem copying file in Operating System - HP-UX

Problem copying file

Olga_1 — Mon, 28 May 2007 10:00:57 GMT

Hello,

I have HP-UX B.11.23 U ia64 server.
There is an intermittent problem to copy file from one FS to another using script.

Manual copy works OK.

Script looks like this and suppose to copy 74 database files:
cp app_data06.dbf /u04/oradata/MSSPROD/app_data06.dbf
cp mssprod_app_data01.dbf /u04/oradata/MSSPROD/mssprod_app_data01.dbf
.... etc

At one point it would exit with Permission denied error. (Permissions and space are not an issue.)
If I check the destination it has already some files copied and one file that looks like half copied:

-rw-r--r-- 1 oracle dba 1572872192 May 25 11:02 mssprod_app_data10.dbf
-rw-r--r-- 1 oracle dba 2097160192 May 25 11:03 mssprod_app_data11.dbf
-rw-r--r-- 1 oracle dba 2097160192 May 25 11:07 mssprod_app_data12.dbf
---------- 1 oracle dba 4718600192 May 25 11:00 mssprod_app_data13.dbf

Any ideas?
Thank you,
Olga

Re: Problem copying file

Andrew Young_2 — Mon, 28 May 2007 10:16:03 GMT

Hi Olga

My guess is that the filesystem you are copying to is NFS mounted. There is a filesize limit of 4GB for an NFS mounted filesystem.

The filesystem copy has stopped but because of the error the permissions have not been set yet.

Regards

Andrew Y

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 10:18:30 GMT

Thank you for the reply, Andrew but I do not copy to NFS. And as I mentioned, if I re-ran the script it works. The problem is intermittent.

Re: Problem copying file

Geoff Wild — Mon, 28 May 2007 10:20:50 GMT

Change script to remove destination files first:

rm /u04/oradata/MSSPROD/*.dbf

then run the copy.

Rgds...Geoff

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 10:23:57 GMT

Geoff, I cleaned up the files before running the script. The destination is empty.

Re: Problem copying file

Geoff Wild — Mon, 28 May 2007 10:30:25 GMT

Okay - need more input please:

bdf /u04/oradata/MSSPROD/

bdf /source

(where source is the source directory)

Is oracle id doing the copying? (I'm assuming as you are not doing a cp -p)

ls -ld /u04/oradata/MSSPROD/

ls -ld /source

Is the db down when you copy the files? (it better be - for you can have issues trying to copy files that are busy).

Rgds...Geoff

Re: Problem copying file

Andrew Young_2 — Mon, 28 May 2007 10:35:40 GMT

Hi Olga

OK. Sorry about that... my bad - its 4TB limit for NFS 4 in any case.

Some question that might point us in the right direction.

Are there any core dumps?

What version of JFS are you running? How recently was it patched?

Is there anything special about the disk or device that you are writing to?

Are there any entries in the /var/adm/syslog/syslog.log?

What user is this running as?
What shell is this running in?
What is the result of the ulimit -a command?

HTH

Andrew Y

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 10:40:53 GMT

Geoff, here it is and thank you for your help.

Destination:
Filesystem kbytes used avail %used Mounted on
/dev/vg03/u04 35848192 25869 33583435 0% /u04

Source
Filesystem kbytes used avail %used Mounted on
/dev/vg03/u77 166920192 156575189 9698470 94% /u77

Is oracle id doing the copying? Yes

ls -ld /u04/oradata/MSSPROD/
drwxr-xr-x 2 oracle dba 1024 May 28 11:08 /u04/oradata/MSSPROD/

ls -ld /source
drwxr-xr-x 2 oracle dba 4096 May 25 13:02 /u77/staging

Is the db down when you copy the files? DB is down

Re: Problem copying file

john korterman — Mon, 28 May 2007 10:52:52 GMT

Hi Olga,

your destination directory was last changed around the time when the file causing the "permission denied" error could have been finished. Are you absolutely sure that noone has tampered with the destination directory permissions?

regards,
John K.

Re: Problem copying file

Geoff Wild — Mon, 28 May 2007 10:55:14 GMT

Only thing that stands out - is the size of /u04 - 33 GB - what is the size of all the db files?

Looks like source is approx 160 GB...

Rgds...Geoff

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 11:04:04 GMT

Andrew, here what you asked for:

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4194300
stack(kbytes) 131072
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

OnlineJFS01 4.1.008

Disks: EVA

User: oracle

SHELL: ksh

No errors in syslog.log?

John, I am sure that nobody changed the permissions.

Geoff,

Files take about 31G, so space should not be an issue.

Thank you all.

Re: Problem copying file

john korterman — Mon, 28 May 2007 11:13:37 GMT

Hi again Olga,

the last modification date of the file mssprod_app_data13.dbf in the destination directory seems to be before the files whose sequence number woukd date them earlier. Are you sure that mssprod_app_data13.dbf in the destination directory does not belong to a failed earlier transfer, and you cannot overwrite it because you do not have the right to do so?

regards,
John K.

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 11:23:42 GMT

Sorry John for the confusion, but in my initial post it is not the real look of the directory when the problem happened. Since it is hard to reproduce it I simulated how directory looked after the script failed. So do not pay attention on time stamps.

Re: Problem copying file

Jov — Mon, 28 May 2007 12:03:49 GMT

Hi Olga,

Have you done a full trace of the script and see what and when it fails on? Also was it consistent with a particular Ora datafile?

set -x the start of the script and log the output to /tmp and see if you can profile the problem?

Jov

Re: Problem copying file

john korterman — Mon, 28 May 2007 12:04:55 GMT

Hi again Olga,

are the permissions displayed correctly? I assume that you will get a "permission denied" error if you as user=oracle tries to overwrite mssprod_app_data13.dbf as is.
A file without permissions is normal in situations when the destination file system ran full during copy. Although you have stated that space is not an issue, please check the syslog for a possible message of af full destination file system.

regards,
John K.

Re: Problem copying file

Andrew Young_2 — Mon, 28 May 2007 13:02:59 GMT

Hi Olga.

Please run ulimit -H -f and ulimit -S -f as the oracle user. This will give the and hard soft limits in terms of bytes being written. The figure given is in 512K blocks.

I'm beginning to think your permission denied error may be caused a user limit or quota condition kicking in at a shell or kernel level preventing you from writing that data.

Because it is a hard/soft limit it is only kicking in intermittently When you rerun the script it ignores the data already copied perhaps.

To change these values to unlimited use the following commands:

ulimit -H -f unlimited
ulimit -S -f unlimited

Note these unlimited settings will affect some of the oracle initialisation scripts since they do not like the unlimited option (unique to the HP ksh)

HTH

Andrew Y

Re: Problem copying file

Olga_1 — Mon, 28 May 2007 13:07:39 GMT

Andrew, thank you very much for not giving up.

Both settings are set to unlimited:

ulimit -H -f
unlimited
ulimit -S -f
unlimited

Re: Problem copying file

Andrew Young_2 — Mon, 28 May 2007 13:11:53 GMT

Hi Olga

That was also not correctly read by me. The file write is already set to unlimited, ignore that posting.

However maybe an increase in the data blocks might make a difference.

Maybe run sar -v 2 120 during the copy to see the effect during the copy. May also use Glance Plus to monitor system activity and utilization during the copy to see if any resources are getting stretched thin at the time

Regards

Andrew Y

Re: Problem copying file

Dennis Handly — Mon, 28 May 2007 18:49:17 GMT

You may want to change your cp to cp -f. That way any leftover files with no permissions won't stop things.

Though you did say you removed the files first.

Re: Problem copying file

Jov — Mon, 28 May 2007 19:31:07 GMT

Is it normal behaviour that destination file gets its permission set after the completion of the copy?

I thought it was dependent on the user and directory settings.

How often does this problem occur, or has it ever completed properly?

Jov