script to trim the file size base on size

[Cheng Wee]

Hi

As above, i don't know how to write the script to trim the file size base on size. Means, let say i want to trim the file from 200mb to 50mb. Any guidance?

Rgs

Cheng Wee

[Ferdie Castro]

Hi Cheng Wee,

Actually the file itself to trim down size must be edited to lessen the size of it. you can use vi then delete some unnecessary lines .
vi filename
then type dd to lines not so important.
then save the file by typing wq! then enter.

Ferdie

[Karthik S S]

Hi Cheng,

You can do it using SAM.

If you are planning to trim the system related log files then the best method is to use sam.

From SAM --> Routine Tasks --> System Log Files

From here either you can trim the file to recommended or manually specified size.

Also you can add custom files to the list. Add whichever file that you want to reduce the size to the list from the Actions menu. Then follow the same procedure for trimming it.

Regards,
Karthik S S

[Cheng Wee]

So is there anyway we can simulate what the sam do using script?

[Elmar P. Kolkman]

SAM does something like this:

cp syslog.log OLDsyslog.log
tail -20 OLDsyslog.log > syslog.log

At least, that's what might do what you want.

[Karthik S S]

I am not sure if you can call this SAM area from with in a script. Is the type of the file that you want to trim is Ascii?? if yes may be we can try to write a script clubbing "wc -l", "du filename" and sed/tail.

We can roughly relate the size of the file with respect to the number of lines. Let us assume that the file size is 10MB and the number of lines are 50000. If you want to trim the size to 5MB then we know that we need approximately delete 25000 lines from the top. Then using tail redirect only the last 25000 lines to a new file and then move it back to overwrite the large original file. We can also use sed for this.

-Karthik S S

[Cheng Wee]

So what if i want to trim it according to size instaed of line number, is there anyway?

[Sanjay Kumar Suri]

head and tail has -c option which can give desired output.

sks

[Sanjay Kumar Suri]

For example:

$tail -c 15 a.sh

While trying same with head, I am finding some error.

sks

[Sanjay Kumar Suri]

Even head is working as the following example shows:

$head -c -n2 a.sh

where -c: The quantity of output is measured in bytes.

sks

[Karthik S S]

with head you need to give as,

head -c -n (byte count) filename

-Karthik S S

[Mark Grant]

If you really,really want a log file to end up an exact number of bytes try something like this. It creates a new file with an extention of ".trim" that should be the exact size you require. You can modify it if you like to delete he original. I have only done a quick test and it doesn't do any error checking so be careful.

run it with "logtrunk.pl

#!/usr/bin/perl

($FILENAME,$NEWSIZE)=@ARGV;
$SIZE=(stat $FILENAME)[7];
$LOSEAMOUNT=$SIZE-$NEWSIZE;

open FILE, "<$FILENAME";
open TRIMEDFILE, ">$FILENAME"."trim";
read FILE, $WASTE, $LOSEAMOUNT;
while(){
print TRIMEDFILE;
}

close FILE;
close TRIMEDFILE;

[Sanjay Kumar Suri]

There is another command split which can give desired result.

$split -b 200 file_name

Check man pages for more options.

sks

[Alan Turner]

Cheng

You can't use "tail" for large files, because it only works in a 20K buffer.

You also need to be careful if applications are holding files open - you don't want to rename the original file, copy the last 50MB back to the orignal file name, then delete the oriignal file, because the applications will continue writing to the original (now "deleted") file, and continue to eat up disk space.

[A. Clay Stephenson · Accepted solution]

Well, to do exactly what you asked, use Perl's truncate function -- on OS's that support the truncate system call -- and HP-UX does:

#!/usr/bin/perl

if ($#ARGV >= 1)
{
$fname = "+< $ARGV[0]";
open FH1, $fname or die "Can't open $ARGV[0]\n";
truncate(FH1,$ARGV[1]);
close FH1;
}

For example,
truncate.pl myfile 5000
would open myfile and then truncate it to exactly 5000 bytes.

This is very useful in cases where you are short of space and cannot copy the contents to another file.

[Rodney Hills]

Since none of the typical file trimming techniques for log files aren't what you are looking for, I'll assume this is a non-ascii work file that has to be a precise size.

You could use "dd" with the count= parameter.

dd bs=1024 count=50k if=oldfile of=newfile

This will created newfile at precisely 50mb. Then you could "mv newfile oldfile" to replace the original file.

HTH

-- Rod Hills

[Cheng Wee]

Hi all,

Sorry for the late reply, been very busy yesterday. Just to clarify one thing, is really work on ascii file, i asking about size is because i wonder how sam can trim log file until recommended size. I interesting with the perl script that can truncate without need to copy the file to another place first and also the concern on not delete the original file. I will look at this in more detail.

Appreciate all replies.

Rgs

Cheng Wee

[Hein van den Heuvel]

Alan Turner wrote on Feb 3, 2004 09:00:47 GMT
>-------------------------------------------------------------------------------
>Cheng

>You can't use "tail" for large files, >because it only works in a 20K buffer.


Nah... that would be soooo seventies!
You can wag that tail well over 20K...
Don't know what the limit (if any) is.


$ uname -a
HP-UX B.11.23 U ia64
$ tail -1000 kc.log > x
$ ls -l x
39059 Feb 4 00:53 x
$ tail -c -10000 kc.log > x
$ ls -l x
10000 Feb 4 00:49 x
$ tail -c -50000 kc.log > x
$ ls -l x
50000 Feb 4 00:49 x

[Cheng Wee]

That is interesting, but my server is still running on hp-ux b.11.00 and it is not working. So is this a new feature on newer O.S?

[Alan Turner]

Hein / Cheng

11.11 seems to be limited to 20K:

cd /var/adm/sw
wc -l swagent.log
8955 swagent.log
tail -4000 swagent.log > /tmp/at
ll /tmp/at
(size = 20480)
wc -l /tmp/at
564 /tmp/at
tail -c -45000 swagent.log > /tmp/atc
ll /tmp/atc
(size = 20480)

also see WARNINGS in 'man tail'

By the way - regarding Clay's response (note his use of "exactly") - this will truncate the file, i.e. chop off the end, and may not be what you want if you would prefer to retain the newest, rather than oldest, records.

One possible approach would be:
a)calculate the file size (ls -s gives it in blocks)
b) work out how many blocks to skip
c) use dd skip= to copy the last 50 MB to a temp file (logtemp)
d) cat logtemp > logfile
e) rm logtemp

The trouble with this approach is that records added after step c) may be lost, or mingle in with the records being copied in from logtemp.

I had a slightly related problem to solve a few months ago, but in my case, the log file was the captured output from a script which spawned a couple of hundred sub-processes. What I did was to replace ">>log" with "|script.pl" where script.pl managed the log file (log and log.OLD) after a certain number of lines had been written.