How to chop off the last part of a large file?

[John Wolfe_1]

Hi:

I need to chop off the last part of a very large file. The file is so large that I can't copy and move it back into place. Does anyone know of a command to allow me to do this to a file without a copy?

TIA,
John

[Steven E. Protter]

The tail command has always worked for me.

tail filename > newfile

SEP

[Umapathy S]

John,
There any many options.

You can use split to split the files into many chunks(files). check man split.

You can view say first 100 lines using head -100 server.out

You can also use tail to see the last part of the file say 100 lines from below tail -100 server.out.

HTH,
Umapathy

[Paul Sperry]

tail -n number_of_lines large_file > new_file

[A. Clay Stephenson]

Because it relies on the underlying truncate() system call which is not implemented in all flavor of UNIX, this is not a universal solution BUT it will work in HP-UX (and most modern UNIX's):

perl -e 'truncate("myfile",10000000)'

This will chop off everything in "myfile" beyond the 10,000,000th byte.

Man 2 truncate for details.

[Elena Leontieva]

I think it should read:

tail -N file > lastN.file

[John Wolfe_1]

Hi Steven:

I guess I was not very clear. I want to keep the first part of a file and discard the last parts without rewriting the file. I would really like to chop a file off at say line 900000 and do the operation "in place".

Thanks,
John

[Jim Mallett]

Clays example will do exactly that. It looks like it's just using a temporary space and replacing the original file. No copy or redirect required.

[Dario_1]

John:

You can use the head command. See the man pages for more information. It does the opposite than the tail command.

head -n 100 filename > newfilname

Regards,

Dario

[A. Clay Stephenson]

Actually, truncate() doesn't use any temporary space. It just goes out to a given offset and WHAM - the ax falls. The remaining data is returned to the filesystem as free space. The tricky part will be converting the start of a line number into an offset - but that ain't so hard either.

[Caesar_3]

Hello!

The perl function truncate will help you
and as A. Clay Stephenson writen that it's
not suporten on all kind of unix systems
you also have library that implement truncate
for c/c++ that you could use not shure about
the name something about filetools.

Caesar

[Jim Mallett]

Thanks for the clarification Clay. I have a few points set aside here next time you respond to one of my threads.

Can't wait to finally get time to open the Perl books.

[Dario_1]

Hi John!

Please disregard my approach since you will have to re-direct to a file.

Regards,

Dario

[curt larson_1]

you can give this a try:

num=9000001 #number of lines to save + 1.

ex -s +"$num,$ d|wq" $yourfile

of course you might want to do some testing to verify there is at least $num lines in your file before doing this. and, I would guess that ex does create a scratch file copy of $yourfile in /var/tmp, so you'll need some free disk space there.

I'm sure there is a way to remove lines in place using perl also.

[H.Merijn Brand (procura · Accepted solution]

Of course the perl example is perfect, but it chops the file after 10_000_000 *bytes*. This example will truncate after 9_000 *lines*:

# perl -e'$f=$ARGV[0];<>for 1..9_000;truncate$f,tell' file_to_truncate

example is for 1 file only!

Enjoy, have FUN! H.Merijn

[John Wolfe_1]

Thanks Procura. You get the 10 points! I couldn't quite figure out how to get the lines converted to bytes. A Clay hinted that it was easy but I guess he was waiting for me to figure it out myself.

Thanks to all,
John

[A. Clay Stephenson]

Actually I'm too dumb to figure out that trick so like a good teacher I leave that as an exercise for the students and let them think I know how to do it.

My second reason was that I question the need for such a command unless one might want to intentionally truncate a log to hide one's tracks in which case he would be well advised to make sure ctime, mtime, and atime are 'fixed' as well.

[Caesar_3]

Hello!

To figure out the size of line you will
need to know the size of line that's mean
scan all file and get size of every line
than in the end truncate what you want to
remove.

Best for you is to made the whole script on perl.

Caesar