topic How to chop off the last part of a large file? in Operating System - HP-UX

How to chop off the last part of a large file?

John Wolfe_1 — Fri, 20 Jun 2003 17:32:40 GMT

Hi:

I need to chop off the last part of a very large file. The file is so large that I can't copy and move it back into place. Does anyone know of a command to allow me to do this to a file without a copy?

TIA,
John

Re: How to chop off the last part of a large file?

Steven E. Protter — Fri, 20 Jun 2003 17:34:31 GMT

The tail command has always worked for me.

tail filename > newfile

SEP

Re: How to chop off the last part of a large file?

Umapathy S — Fri, 20 Jun 2003 17:36:37 GMT

John,
There any many options.

You can use split to split the files into many chunks(files). check man split.

You can view say first 100 lines using head -100 server.out

You can also use tail to see the last part of the file say 100 lines from below tail -100 server.out.

HTH,
Umapathy

Re: How to chop off the last part of a large file?

Paul Sperry — Fri, 20 Jun 2003 17:36:47 GMT

tail -n number_of_lines large_file > new_file

Re: How to chop off the last part of a large file?

A. Clay Stephenson — Fri, 20 Jun 2003 17:38:34 GMT

Because it relies on the underlying truncate() system call which is not implemented in all flavor of UNIX, this is not a universal solution BUT it will work in HP-UX (and most modern UNIX's):

perl -e 'truncate("myfile",10000000)'

This will chop off everything in "myfile" beyond the 10,000,000th byte.

Man 2 truncate for details.

Re: How to chop off the last part of a large file?

Elena Leontieva — Fri, 20 Jun 2003 17:38:57 GMT

I think it should read:

tail -N file > lastN.file

Re: How to chop off the last part of a large file?

John Wolfe_1 — Fri, 20 Jun 2003 17:42:10 GMT

Hi Steven:

I guess I was not very clear. I want to keep the first part of a file and discard the last parts without rewriting the file. I would really like to chop a file off at say line 900000 and do the operation "in place".

Thanks,
John

Re: How to chop off the last part of a large file?

Jim Mallett — Fri, 20 Jun 2003 17:52:06 GMT

Clays example will do exactly that. It looks like it's just using a temporary space and replacing the original file. No copy or redirect required.

Re: How to chop off the last part of a large file?

Dario_1 — Fri, 20 Jun 2003 18:01:54 GMT

John:

You can use the head command. See the man pages for more information. It does the opposite than the tail command.

head -n 100 filename > newfilname

Regards,

Dario

Re: How to chop off the last part of a large file?

A. Clay Stephenson — Fri, 20 Jun 2003 18:02:26 GMT

Actually, truncate() doesn't use any temporary space. It just goes out to a given offset and WHAM - the ax falls. The remaining data is returned to the filesystem as free space. The tricky part will be converting the start of a line number into an offset - but that ain't so hard either.

Re: How to chop off the last part of a large file?

Caesar_3 — Fri, 20 Jun 2003 18:06:04 GMT

Hello!

The perl function truncate will help you
and as A. Clay Stephenson writen that it's
not suporten on all kind of unix systems
you also have library that implement truncate
for c/c++ that you could use not shure about
the name something about filetools.

Caesar

Re: How to chop off the last part of a large file?

Jim Mallett — Fri, 20 Jun 2003 18:13:55 GMT

Thanks for the clarification Clay. I have a few points set aside here next time you respond to one of my threads.

Can't wait to finally get time to open the Perl books.

Re: How to chop off the last part of a large file?

Dario_1 — Fri, 20 Jun 2003 18:17:38 GMT

Hi John!

Please disregard my approach since you will have to re-direct to a file.

Regards,

Dario

Re: How to chop off the last part of a large file?

curt larson_1 — Fri, 20 Jun 2003 18:21:52 GMT

you can give this a try:

num=9000001 #number of lines to save + 1.

ex -s +"$num,$ d|wq" $yourfile

of course you might want to do some testing to verify there is at least $num lines in your file before doing this. and, I would guess that ex does create a scratch file copy of $yourfile in /var/tmp, so you'll need some free disk space there.

I'm sure there is a way to remove lines in place using perl also.

Re: How to chop off the last part of a large file?

H.Merijn Brand (procura — Fri, 20 Jun 2003 18:39:35 GMT

Of course the perl example is perfect, but it chops the file after 10_000_000 *bytes*. This example will truncate after 9_000 *lines*:

# perl -e'$f=$ARGV[0];<>for 1..9_000;truncate$f,tell' file_to_truncate

example is for 1 file only!

Enjoy, have FUN! H.Merijn

Re: How to chop off the last part of a large file?

John Wolfe_1 — Fri, 20 Jun 2003 19:06:31 GMT

Thanks Procura. You get the 10 points! I couldn't quite figure out how to get the lines converted to bytes. A Clay hinted that it was easy but I guess he was waiting for me to figure it out myself.

Thanks to all,
John

Re: How to chop off the last part of a large file?

A. Clay Stephenson — Fri, 20 Jun 2003 19:12:37 GMT

Actually I'm too dumb to figure out that trick so like a good teacher I leave that as an exercise for the students and let them think I know how to do it.

My second reason was that I question the need for such a command unless one might want to intentionally truncate a log to hide one's tracks in which case he would be well advised to make sure ctime, mtime, and atime are 'fixed' as well.

Re: How to chop off the last part of a large file?

Caesar_3 — Fri, 20 Jun 2003 19:33:47 GMT

Hello!

To figure out the size of line you will
need to know the size of line that's mean
scan all file and get size of every line
than in the end truncate what you want to
remove.

Best for you is to made the whole script on perl.

Caesar