topic Re: tail limit in Operating System - HP-UX

tail limit

jerry1 — Fri, 05 Aug 2005 14:09:53 GMT

Has anyone had a problem with not being able
to tail a set number of lines.

tail -10000

Re: tail limit

Srini Jay — Fri, 05 Aug 2005 14:18:39 GMT

To my knowledge tail gets you upto a max of 20 kilobytes of data from end of file irrespective of number of lines.

So if the bytes used by last 10000 of the file is less than 20Kb, then the command will fetch you all 10000 lines or else just the last 20kb of data.

Re: tail limit

Mel Burslan — Fri, 05 Aug 2005 17:20:09 GMT

when working with files with huge number of lines, it is always better to use some sed magic.

instead of

tail -10000

file=whatever
fileLEN=`cat $file|wc -l`
(( trim_from_top=$fileLEN-10000 ))
if [ $trim_from_top -gt 0 ]
then
sed -e "1,${trim_from_top}d" $file >tail1000.out
else
echo "Your file does not have 10000 lines in it"
fi

I think this script is self explanatory.

Re: tail limit

Marlou Everson — Tue, 16 Aug 2005 17:45:42 GMT

Under 11.11, there is a tail(1) patch which addresses this problem: PHCO_27138 (most current, 3 star rating).

Marlou

Part of readme text:
Symptom:
tail(1) command does not display more than 20 kilo bytes relative to the end of file.

Defect Description:
The number of line in the output of the tail(1) command is limited by the 20KB buffer size.
For example:
$ ll Filename
-r--r--r-- 1 bsanjay uxdev 40098 Nov 1 17:37 Filename
$ /usr/bin/tail -c 30000 Filename | wc -c
20480
$ cat Filename | /usr/bin/tail -c 30000 Filename | wc -c
20480

Resolution:
The design of the tail(1) command is modified. The following are the new features available with this patch:
- When the input is not a pipe, there is no limit on the output from tail(1). Even large files(>2GB) can be used as input to tail(1) command. In otherwords, all options to tail(-c, -n, -b) take a value greater than 2GB.
For example:
$ ll Filename
-r--r--r-- 1 bsanjay uxdev 2150000001 Nov 1 17:37 Filename
$ /usr/bin/tail -c 2150000000 Filename | wc -c
2150000000

- When the input to tail(1) is through a pipe, the new tail(1) command uses dynamically allocated buffers. Because of dynamic memory allocation, the output of tail(1) is limited to process limits. This limit is much higher than the old 20 KB limit.
For example:
$ ll Filename
-r--r--r-- 1 bsanjay uxdev 2000000 Nov 1 17:37 Filename
$ cat Filename | /usr/bin/tail -c 1000000 | wc -c
1000000

Note: If the output from the above example is less than the expected output, it is an indication that tail(1) is unable allocate more memory and the process limit is reached.