cron job every minute increases root file size, it is going to be full if continues

[George_206]

Dear all,
I have set the following cron job to run every minute.
0 9 * * * /pp3/script_del > /pp3/out 2>&1

But I see that it allocates disk space in root file system. So if it runs for a week root file system is going to be full. But I cannot see how to resolve this.

Can you help me to find out why ?

Thanks.
George.

[Stuart Browne]

.. what does this script *do* ?

Besides, that doesn't run every minute, that runs at 9am (once a day).

[George_206]

Yes, you are right. I posted the wrong job.
The correct one is * * * * * /pp3/fr_script1 > /pp3/cron_out 2>&1

Actually the fr_script1 monitors web server logs, and searches for some pattern (eg. ip). If it finds any matching it forms an SQL INSERT statement an runs sqlplus in silent mode to update a DB.

[George_206]

In addition it is run by root. If I set it to be run by another user, will it help?

[George_206]

In addition the job is run by root. If I set it to run by another user will it help?

[Stuart Browne]

So it extracts some part of the log files, insert it into a database, so.. why keep the the logs that get inserted?

You've got a few options.

1) Compress the extracted information after you're done with it (inserted into the db).

2) Delete them once you're done with the details

3) don't do it every minute, and rotate/compress your web logs every day.

[Vitaly Karasik_1]

>But I see that it allocates disk space in root file system

how do you see this?

Is /pp3/out file growing? it shouldn't.

[George_206]

Actually the /dev/vg00/lvol3 is increasing (from 53% ->62% after running the job for a few days).
The problem is that lvol3 cannot be easily extended due to

lvextend error.not enough free physical extends


Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol3 102400 63590 38624 62% 3076 9700 24% /
/dev/vg00/lvol1 99669 37314 52388 42% 49 16079 0% /stand
/dev/vg00/lvol7 2097152 1122939 913585 55% 40434 243550 14% /var
/dev/vg00/lvol6 2097152 926816 1097227 46% 23949 292583 8% /usr
/dev/vg00/lvol5 307200 72329 220734 25% 1193 58715 2% /tmp
/dev/vg02/pp3test 2560000 1634915 867623 65% 10868 231268 4% /pp3test
/dev/vg02/pp3prod 4096000 2805417 1210344 70% 6024 322644 2% /pp3prod
/dev/vg00/oracle 5120000 3109346 1890729 62% 58703 502661 10% /oracle
/dev/vg00/lvol4 1048576 699947 326855 68% 9156 87156 10% /opt
/dev/vg00/lvol8 20480 2203 17197 11% 295 4569 6% /home

[Vitaly Karasik_1]

>Actually the /dev/vg00/lvol3 is increasing >(from 53% ->62% after running the job for a >few days).

OK, it is increasing.
But why do you think that it is *because* this cronjob?!

I suggest you to monitor (using crontab):
1) size of /pp3/out file
2) du /pp3 |sort -rn
3) find /pp3 -mtime -1

[George_206]

Well it seems that has to do with i-nodes, since avail inodes are also reduced. I suspect that the script leaves some open files(?).


$ bdf -i
Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol3 102400 63605 38610 62% 3076 9696 24% /
/dev/vg00/lvol1 99669 37314 52388 42% 49 16079 0% /stand
/dev/vg00/lvol7 2097152 1122939 913585 55% 40434 243550 14% /var
/dev/vg00/lvol6 2097152 926816 1097227 46% 23949 292583 8% /usr
/dev/vg00/lvol5 307200 72329 220734 25% 1193 58715 2% /tmp
/dev/vg02/pp3test 2560000 1634915 867623 65% 10868 231268 4% /pp3test
/dev/vg02/pp3prod 4096000 2805919 1209874 70% 6021 322519 2% /pp3prod
/dev/vg00/oracle 5120000 3109346 1890729 62% 58703 502661 10% /oracle
/dev/vg00/lvol4 1048576 699947 326855 68% 9156 87156 10% /opt
/dev/vg00/lvol8 20480 2203 17197 11% 295 4569 6% /home
$

[Vitaly Karasik_1]

in short, this crontab line seems OK to me.
What do you have inside of script_del?

which files do you have into lvol3? oracle DBs, logs, indexes?

check open files by

lsof |grep pp3

[George_206]

lsof is not installed and I cannot install it

[Vitaly Karasik_1]

OK, let's continue without lsof.

start monitor pps3 filesystem as I suggested.

In addition - I guess that "script_del" is cleaning out old files/logs.
How you do it? You should be aware that you cannot just remove files which is opened by some running process. In this case application is continues to write to the inode even file is deleted.

Can you compare and send us output of "du -s /pp3" and "df /pp3"?

[George_206]

Actually a part of the script is below. It is not the most efficient but it is a first approach.

In brief, I have an input file with some records which may occur many times. What I want is to remove every line which is more than three time in the input file and create a new file with the deleted record.

I have thus created the following script by creating some temp files. Perhaps this is not the correct way to handle temp files.


#count number of string occurence in CA file
#and if > than 3 create a record in CA1 file

sort -k 1 /pp3/input >> /pp3/input.sorted
uniq -c /pp3/input.sorted >> /pp3/input.sorted.uniq

awk '{if ($1>2) print $2}' /pp3/input.sorted.uniq >> /pp3test/output.uniq

awk '{print $2}' /pp3/input.sorted.uniq >> /pp3/input.sorted.uniq1

awk -F"|" '
BEGIN {
while( getline < "/pp3/output.uniq" )
{ arr[$0]=1 }
}
arr[$0] != 1 { print }
' /pp3/input.sorted > /pp3/input.sorted1

rm /pp3/input.sorted.uniq
rm /pp3/input.sorted.uniq
rm /pp3/input.sorted
mv /pp3/input.sorted1 /pp3test/input.sorted

[Vitaly Karasik_1]

it seems OK.

but anyway - can you check big & new files on /pp3 ?

[James R. Ferguson · Accepted solution]

Hi George:

You have posted this thread in BOTH HP-UX and LINUX! That's not helpful and wastes effort:

http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1196439

Regards!

...JRF...

[George_206]

Yes, you are correct.
The problem is that I intially posted in the Linux (but I wanted hp-ux) but I could not find it through the search, so I thought it was not actually posted.

But anyway, I think I have finally found the cause of the problem.

The problem is that in the HP9000 the audit option was ON, so the audit files were increasing every time the cron job was executed. Since the job includes several system commands, which are monitored by the audit, the problem was that a week after the root disk space (where the audit log was located) was significantly minimized.

So I turned the audit OFF and rotated the logs and now it seems that everything is OK.

Thanks everyone for your effort and immediate resonse. I do appreciate.

Now it's time to assign points!!!

Thanks.
George.

[George_206]

The problem is that in the HP9000 the audit option was ON, so the audit files were increasing every time the cron job was executed. Since the job includes several system commands, which are monitored by the audit, the problem was that a week after the root disk space (where the audit log was located) was significantly minimized.

So I turned the audit OFF and rotated the logs and now it seems that everything is OK.