topic Re: script to trim the file size base on size in Operating System - HP-UX

script to trim the file size base on size

Cheng Wee — Mon, 02 Feb 2004 22:35:36 GMT

Hi

As above, i don't know how to write the script to trim the file size base on size. Means, let say i want to trim the file from 200mb to 50mb. Any guidance?

Rgs

Cheng Wee

Re: script to trim the file size base on size

Ferdie Castro — Mon, 02 Feb 2004 22:53:23 GMT

Hi Cheng Wee,

Actually the file itself to trim down size must be edited to lessen the size of it. you can use vi then delete some unnecessary lines .
vi filename
then type dd to lines not so important.
then save the file by typing wq! then enter.

Ferdie

Re: script to trim the file size base on size

Karthik S S — Mon, 02 Feb 2004 23:13:37 GMT

Hi Cheng,

You can do it using SAM.

If you are planning to trim the system related log files then the best method is to use sam.

From SAM --> Routine Tasks --> System Log Files

From here either you can trim the file to recommended or manually specified size.

Also you can add custom files to the list. Add whichever file that you want to reduce the size to the list from the Actions menu. Then follow the same procedure for trimming it.

Regards,
Karthik S S

Re: script to trim the file size base on size

Cheng Wee — Tue, 03 Feb 2004 00:55:32 GMT

So is there anyway we can simulate what the sam do using script?

Re: script to trim the file size base on size

Elmar P. Kolkman — Tue, 03 Feb 2004 01:11:40 GMT

SAM does something like this:

cp syslog.log OLDsyslog.log
tail -20 OLDsyslog.log > syslog.log

At least, that's what might do what you want.

Re: script to trim the file size base on size

Karthik S S — Tue, 03 Feb 2004 01:16:11 GMT

I am not sure if you can call this SAM area from with in a script. Is the type of the file that you want to trim is Ascii?? if yes may be we can try to write a script clubbing "wc -l", "du filename" and sed/tail.

We can roughly relate the size of the file with respect to the number of lines. Let us assume that the file size is 10MB and the number of lines are 50000. If you want to trim the size to 5MB then we know that we need approximately delete 25000 lines from the top. Then using tail redirect only the last 25000 lines to a new file and then move it back to overwrite the large original file. We can also use sed for this.

-Karthik S S

Re: script to trim the file size base on size

Cheng Wee — Tue, 03 Feb 2004 01:16:27 GMT

So what if i want to trim it according to size instaed of line number, is there anyway?

Re: script to trim the file size base on size

Sanjay Kumar Suri — Tue, 03 Feb 2004 01:23:51 GMT

head and tail has -c option which can give desired output.

sks

Re: script to trim the file size base on size

Sanjay Kumar Suri — Tue, 03 Feb 2004 01:28:51 GMT

For example:

$tail -c 15 a.sh

While trying same with head, I am finding some error.

sks

Re: script to trim the file size base on size

Sanjay Kumar Suri — Tue, 03 Feb 2004 01:32:07 GMT

Even head is working as the following example shows:

$head -c -n2 a.sh

where -c: The quantity of output is measured in bytes.

sks

Re: script to trim the file size base on size

Karthik S S — Tue, 03 Feb 2004 01:32:24 GMT

with head you need to give as,

head -c -n (byte count) filename

-Karthik S S

Re: script to trim the file size base on size

Mark Grant — Tue, 03 Feb 2004 02:02:04 GMT

If you really,really want a log file to end up an exact number of bytes try something like this. It creates a new file with an extention of ".trim" that should be the exact size you require. You can modify it if you like to delete he original. I have only done a quick test and it doesn't do any error checking so be careful.

run it with "logtrunk.pl

#!/usr/bin/perl

($FILENAME,$NEWSIZE)=@ARGV;
$SIZE=(stat $FILENAME)[7];
$LOSEAMOUNT=$SIZE-$NEWSIZE;

open FILE, "<$FILENAME";
open TRIMEDFILE, ">$FILENAME"."trim";
read FILE, $WASTE, $LOSEAMOUNT;
while(){
print TRIMEDFILE;
}

close FILE;
close TRIMEDFILE;

Re: script to trim the file size base on size

Sanjay Kumar Suri — Tue, 03 Feb 2004 02:04:47 GMT

There is another command split which can give desired result.

$split -b 200 file_name

Check man pages for more options.

sks

Re: script to trim the file size base on size

Alan Turner — Tue, 03 Feb 2004 04:00:47 GMT

Cheng

You can't use "tail" for large files, because it only works in a 20K buffer.

You also need to be careful if applications are holding files open - you don't want to rename the original file, copy the last 50MB back to the orignal file name, then delete the oriignal file, because the applications will continue writing to the original (now "deleted") file, and continue to eat up disk space.

Re: script to trim the file size base on size

A. Clay Stephenson — Tue, 03 Feb 2004 09:23:48 GMT

Well, to do exactly what you asked, use Perl's truncate function -- on OS's that support the truncate system call -- and HP-UX does:

#!/usr/bin/perl

if ($#ARGV >= 1)
{
$fname = "+< $ARGV[0]";
open FH1, $fname or die "Can't open $ARGV[0]\n";
truncate(FH1,$ARGV[1]);
close FH1;
}

For example,
truncate.pl myfile 5000
would open myfile and then truncate it to exactly 5000 bytes.

This is very useful in cases where you are short of space and cannot copy the contents to another file.

Re: script to trim the file size base on size

Rodney Hills — Tue, 03 Feb 2004 09:35:26 GMT

Since none of the typical file trimming techniques for log files aren't what you are looking for, I'll assume this is a non-ascii work file that has to be a precise size.

You could use "dd" with the count= parameter.

dd bs=1024 count=50k if=oldfile of=newfile

This will created newfile at precisely 50mb. Then you could "mv newfile oldfile" to replace the original file.

HTH

-- Rod Hills

Re: script to trim the file size base on size

Cheng Wee — Tue, 03 Feb 2004 19:45:57 GMT

Hi all,

Sorry for the late reply, been very busy yesterday. Just to clarify one thing, is really work on ascii file, i asking about size is because i wonder how sam can trim log file until recommended size. I interesting with the perl script that can truncate without need to copy the file to another place first and also the concern on not delete the original file. I will look at this in more detail.

Appreciate all replies.

Rgs

Cheng Wee

Re: script to trim the file size base on size

Hein van den Heuvel — Wed, 04 Feb 2004 01:01:15 GMT

Alan Turner wrote on Feb 3, 2004 09:00:47 GMT
>-------------------------------------------------------------------------------
>Cheng

>You can't use "tail" for large files, >because it only works in a 20K buffer.

Nah... that would be soooo seventies!
You can wag that tail well over 20K...
Don't know what the limit (if any) is.

$ uname -a
HP-UX B.11.23 U ia64
$ tail -1000 kc.log > x
$ ls -l x
39059 Feb 4 00:53 x
$ tail -c -10000 kc.log > x
$ ls -l x
10000 Feb 4 00:49 x
$ tail -c -50000 kc.log > x
$ ls -l x
50000 Feb 4 00:49 x

Re: script to trim the file size base on size

Cheng Wee — Wed, 04 Feb 2004 01:20:03 GMT

That is interesting, but my server is still running on hp-ux b.11.00 and it is not working. So is this a new feature on newer O.S?

Re: script to trim the file size base on size

Alan Turner — Wed, 04 Feb 2004 05:11:50 GMT

Hein / Cheng

11.11 seems to be limited to 20K:

cd /var/adm/sw
wc -l swagent.log
8955 swagent.log
tail -4000 swagent.log > /tmp/at
ll /tmp/at
(size = 20480)
wc -l /tmp/at
564 /tmp/at
tail -c -45000 swagent.log > /tmp/atc
ll /tmp/atc
(size = 20480)

also see WARNINGS in 'man tail'

By the way - regarding Clay's response (note his use of "exactly") - this will truncate the file, i.e. chop off the end, and may not be what you want if you would prefer to retain the newest, rather than oldest, records.

One possible approach would be:
a)calculate the file size (ls -s gives it in blocks)
b) work out how many blocks to skip
c) use dd skip= to copy the last 50 MB to a temp file (logtemp)
d) cat logtemp > logfile
e) rm logtemp

The trouble with this approach is that records added after step c) may be lost, or mingle in with the records being copied in from logtemp.

I had a slightly related problem to solve a few months ago, but in my case, the log file was the captured output from a script which spawned a couple of hundred sub-processes. What I did was to replace ">>log" with "|script.pl" where script.pl managed the log file (log and log.OLD) after a certain number of lines had been written.