topic Re: Compressing files in Operating System - HP-UX

Compressing files

Aziz Zouagui — Mon, 13 May 2002 19:58:25 GMT

Greetings,

I have and oracle database 8.1.7.3 that generates a lot of archive logs, each one is 200 Mb. I have croned a job every 15 minutes that checks the directory and if it finds any new files, it will compress them.

Now, here is the question:

If the Oracle Archiver process happens to start archiving a file but has not yet completed writing the entire content of the file to disk, if my croned compress job kicks in, will it be able to compress that file even if Oracle has not finished with it yet ? I am a little paranoid, because if the system lets the compress to go through, then I may have corrupted the archive logs.

In other words, does HP-UX prevent other processes from accessing the file ? Bear in mind that compress reads the file and writes to a new file with *.Z and removes the original one.

Any thoughts would be helpful.

Thank you.-

Re: Compressing files

A. Clay Stephenson — Mon, 13 May 2002 20:00:20 GMT

The simple answer is no; that is up to you.

Re: Compressing files

Duncan Edmonstone — Mon, 13 May 2002 20:04:52 GMT

You can of course use the fuser command to see if oracle stil has the file open. Once oracle has finished writing out the file it doesn't open it again so you can safely compress it.

HTH

Duncan

Re: Compressing files

A. Clay Stephenson — Mon, 13 May 2002 20:08:01 GMT

One thing that you could do is determine if a file has been modified within some period of time. If not then it is safe to compress it. Here is one method:

FNAME="myfile"
SECONDS=600
fileage.pl -m -s ${SECONDS} ${FNAME}
STAT=$?
if [ ${STAT} -eq 0 ]
then
echo "File: ${FNAME} has not been modified in the last ${SECONDS} seconds; it is safe to compress."
else
echo "File ${FNAME} has been modified; not safe to compress."
fi

I've attached fileage.pl; invoke fileage.pl without arguments for full usage.

Re: Compressing files

John Carr_2 — Mon, 13 May 2002 20:45:07 GMT

Hi

I have just read the man page for compress and it says this:

Access Control Lists
compress retains a file's access control list when compressing and expanding data.

I think this is saying it basically locks the file could one of the guru's comment on this.

cheers
John.

Re: Compressing files

A. Clay Stephenson — Mon, 13 May 2002 20:53:08 GMT

Sorry, that dog won't hunt. ACL's are a set of extended permisssions. All that is saying is that both the compressed and uncompressed forms of the file retain the original's ACL's (if supported by the filesystem and provided that the ACL's have been set for the file in question.)

Re: Compressing files

harry d brown jr — Mon, 13 May 2002 20:56:36 GMT

John,

If you are compressing FILEA and some process is still writing to it, and the compress completes, then the original file is DELETED, meaning the process still writing to FILEA won't be writing to squat (actually never-never-land).

live free or die
harry

Re: Compressing files

Aziz Zouagui — Mon, 13 May 2002 21:33:35 GMT

Hey guys,

I appreciate all the input I am getting.
I like the fuser suggestion, I will have the job check the fuser before compressing the file, if there is a process list next to the file, then either sleep for a while and try again or abort to the next file and come back to the busy one later.

Please if there is a better suggestion, do not hesitate to update the forum.

Thank you for the feedback.

Re: Compressing files

Jeff Schussele — Tue, 14 May 2002 01:02:50 GMT

Hi Aziz,

I think the question boils down to...Why are you compressing them?
If it's disk space...Then as Clay puts it "That's dog's hunting way too much!".
In that case you really need to extend the VG to hold proper log count/length/size.
If it's to backup & save tape space then I'd recommend you mirror the VG & lvsplit the mirror out prior to compressing & archiving then lvmerge it back in. If redundancy is a major concern & you're nervous about having just one copy for *any* time period then keep 2 mirror copies & you'll always have at least 2 copies on-line.
If you just want to "roll" the logs, I believe Oracle handles that as well. I think you can mv the file to a relevant, time-stamped filename prior to compressing it & Oracle will start a new log...mind you I'm not an Oracle expert but I'm sure one will correct me if I'm mistaken.

Rgds,
Jeff

Re: Compressing files

John Carr_2 — Tue, 14 May 2002 05:49:27 GMT

Hi

as A.Clay and Harry make clear the file is not locked presumably even if you fuser before start of compress another process can effectively attach to file during compress and think it is writing to it.

I would be tempted to rename the file before compression commences to ensure this cannot happen.

As Jeff points out the real solution is to pick another alternative.

cheers
John.

Re: Compressing files

Bruno Cunha — Tue, 14 May 2002 07:22:58 GMT

Hello, you can compress all the files except the last one, in this way if Oracle is still writing to the file you don't compress it, or, you can check the size of each file before compression, if the file is in the maximum size (200mb in your case) then Oracle as finish writing.

Bruno Cunha

Re: Compressing files

Jean-Luc Oudart — Tue, 14 May 2002 08:46:16 GMT

fuser -fu will give list of processes against this file.

In case of recovery, you would have to uncompress the file(s) as they're requested.
This might be a bit tricky if you don't have the space for it !

My opinion would be that you should have the space for the archive log files.

Re: Compressing files

Bill Thorsteinson — Tue, 14 May 2002 13:00:51 GMT

You could use find with a
timestamp file to compress
all uncomressed files older
than a timestamp file.
Something like:

touch compress.$$

sleep 30

find . -name arch\*.arc ! -newer compress.$$ -exec compress {} \;

rm compress.$$

Re: Compressing files

Aziz Zouagui — Tue, 14 May 2002 13:38:32 GMT

Well, I found a much safer solution that I tested and it works great.

First the cron job gets a list of all the files to compress then I check each file against the v$archived_log table in oracle.

If the file is properly archived and oracle is done with it, the Archiver process makes an entry into this table, making it safe to move or compress the file.

I have tested this by issuing an "Alter system switch logfile" right after that I started watching the directory as the file makes to the archive directory and at the same time I query the V$Archived_log table for the existence of the file. The table never showed the file until oracle finished archiving then an entry popped up in the table.

so the cron job never touches a file if it is not in the V$ARCHIVED_LOG table. If anyone is interested in the script, I will be glad to send it to them.

By the way, to answer the other questions why not to give enough space. This database generates close to 50 Gb of archive logs a day.
The directory that holds all of this is 40 Gb. Compressing and sending the files to a stand by database database in another server and eventually to tape, is the best way to have space available for the next day.

Again, many thanks to everyone who contributed their time and intellect to assist in solving this problem

Thank you.

Re: Compressing files

Thomas Yake — Wed, 15 May 2002 12:08:19 GMT

Regardless of your process issues, you might want to replace the compress command with gzip:

itax_dev /tmp/algo # cp mail.log mail.log.preZ.txt
itax_dev /tmp/algo # cp mail.log mail.log.pregz.txt
itax_dev /tmp/algo # ls -arltp
total 7698
drwxrwxrwx 19 bin bin 6144 May 15 09:07 ../
-r--r----- 1 ty hpadmin 1310919 May 15 09:07 mail.log
-r--r----- 1 ty hpadmin 1310919 May 15 09:07 mail.log.preZ.txt
drwxr-x--- 2 ty hpadmin 96 May 15 09:07 ./
-r--r----- 1 ty hpadmin 1310919 May 15 09:07 mail.log.pregz.txt
itax_dev /tmp/algo # gzip mail.log.pregz.txt
itax_dev /tmp/algo # compress mail.log.preZ.txt
itax_dev /tmp/algo # lt
total 3168
drwxrwxrwx 19 bin bin 6144 May 15 09:07 ../
-r--r----- 1 ty hpadmin 1310919 May 15 09:07 mail.log
-r--r----- 1 ty hpadmin 203755 May 15 09:07 mail.log.preZ.txt.Z
-r--r----- 1 ty hpadmin 98570 May 15 09:07 mail.log.pregz.txt.gz
drwxr-x--- 2 ty hpadmin 1024 May 15 09:08 ./
itax_dev /tmp/algo

Re: Compressing files

Aziz Zouagui — Wed, 15 May 2002 12:40:10 GMT

Great info, you mean gzip compresses data much more than compress ?

I didn't know that, I thought they are almost the same. But, clearly from your example, it looks like the gzip condensed more data than compress did. I will definitely have to switch from compress to gzip.

Great input.