topic Re: grep: not enough memory ??? in Operating System - HP-UX

grep: not enough memory ???

someone_4 — Wed, 27 Feb 2002 03:58:47 GMT

Hello everyone ..
seems I am starting eat live and dream HPUX .. fun huh ..
Well we are trying to cat and grep a huge file.
154082574 Feb 26 21:50 radius.log
When I try to grep .. I get this error
cat radius.log | grep spr8345409a1
grep: not enough memory

I can grep a smaller file but I cant get that big file. Does grep have limits? Any other way to look for a string in a huge file?

Thanks,
Richard

Re: grep: not enough memory ???

Steven Sim Kok Leong — Wed, 27 Feb 2002 04:02:19 GMT

Hi,

Try this perl script:

#!/usr/bin/perl

open (FILE, "radius.log");

while ()
{
if /spr8345409a1/
{
print "$_\n";
}
}

Btw, what is your maxdsiz and ulimit?

Hope this helps. Regards.

Steven Sim Kok Leong

Re: grep: not enough memory ???

someone_4 — Wed, 27 Feb 2002 04:20:56 GMT

maxdsiz
0x04000000
Calculated Value: 67108864

ulimit -d
65536

Do I need to bump ulimit?
And will a cat file | grep whatever
use allot of cpu ?

Richard

Re: grep: not enough memory ???

Steven Sim Kok Leong — Wed, 27 Feb 2002 04:55:54 GMT

Hi,

Does the perl script work for you? If it worked, then it was probably a grep limitation.

Hope this helps. Regards.

Steven Sim Kok Leong

Re: grep: not enough memory ???

SHABU KHAN — Wed, 27 Feb 2002 05:39:44 GMT

Hi,

I would first try:
grep "spr8345409a1" radius.log

If this comes back with "not enough memory" then try feeding it to xargs like this:

ls radius.log|xargs grep "spr8345409a1"

Thanks,
Shabu

Re: grep: not enough memory ???

someone_4 — Wed, 27 Feb 2002 15:19:39 GMT

no the perl script spit out some errors.

syntax error at ./perl.scr line 5, near "if /spr8345409a1/"
syntax error at ./perl.scr line 9, near "}"
Execution of ./perl.scr aborted due to compilation errors.

Richard

Re: grep: not enough memory ???

Steve Steel — Wed, 27 Feb 2002 15:21:31 GMT

Hi

This could be swap so check your syslog.

Also try the old do loop to slow it down

cat file|while read line
do
echo $line|grep spr8345409a1
done

Steve Steel

Re: grep: not enough memory ???

Rodney Hills — Wed, 27 Feb 2002 15:23:39 GMT

To fix perl script, change "if" to-

if ( /spr8345409a1/ )

-- Rod Hills

Re: grep: not enough memory ???

someone_4 — Wed, 27 Feb 2002 15:29:55 GMT

well ..
Here is what the perl script gave me ..

./perl.scr
Out of memory during "large" request for 67112960 bytes at ./perl.scr line 3.

Re: grep: not enough memory ???

Rodney Hills — Wed, 27 Feb 2002 15:38:39 GMT

This out of memory condition might be caused by no ^j (new line) in the file. grep and perl are record processing utilities.

If this is the case, you can use perl to do binary reads, reading in a block at a time and scanning that block.

-- Rod Hills

Re: grep: not enough memory ???

Eric Ladner — Wed, 27 Feb 2002 15:45:52 GMT

You're maxdsize is pretty small (still at the default, I believe), although it seems strange that grep would be causing those kinds of problems.

Is this a binary file? Grep is suppposed to be line oriented and I know I have grepped things out of files in the 1 to 2 gig range, but they were text files with newline characters embedded.

Re: grep: not enough memory ???

Frank Slootweg — Thu, 28 Feb 2002 10:36:34 GMT

If this is on HP-UX 10.20, then there are (customer) reports that grep patch PHCO_13451 *might* help. However your scenario is not specifically mentioned in the .txt file of the patch.

Other reports indicate that maxdsiz should be greater than the file's size, which, as far as I can tell is not the case in your case.

Yet other reports use split(1) to split the file into smaller pieces and grep the pieces. May be you can use this trick to see if maxdsiz is indeed the likely culprit.

Just to be sure: This is a *text* file, right? I.e. a file with lines ending with a newline character and with line lengths less than 2048 bytes (see glossary(9) and limits(5)). If not,you may want to use fold(1).

Re: grep: not enough memory ???

H.Merijn Brand (procura — Thu, 28 Feb 2002 13:19:43 GMT

http://hpux.tn.tudelft.nl/hppd/hpux/Gnu/grep-2.5a/ is the GNU version of grep, which might help you out.

With all those memory requirements passing by, a question might be apropriate: Are the `lines' in radius.log separated by newlines, or by something else? If there are no newlines, most unix utilities will fail, because they work line based.

GNU grep works block based, and perl works line based by default, but can easily be told to work block based or accept any record separator you like.

with perl you might get better results in setting the input record separator to something /not/ in the pattern. But that still leaves the question: what should be printed if the lines are not newline separated?

perl -ne '/spr8345409a1/ and print' radius.log

for a much simpler version than all perl examples given, and a safer version could be

Steven's version has some bugs (apart from the missing () in the if):

#!/usr/bin/perl

open (FILE, "radius.log");

while () {
if (/spr8345409a1/) {
print "$_\n";
}
}

1. open FILE, "< radius.log" or die "radius.log: $!";
always check the return status of the open, reading from a not opened handle might be surprising
always open files for reading using "<" in front to prevent strange and unwanted effects:

my $x = "radius.log"
open LOGFILE, $x;

care to think what would happen if $x came from the outside world and contains "| xargs rm -rf /"?

2. print;

perl reads /lines/ separated by newlines (at least on unix) but does not strip them, so $_ still has the newline at the end. $_ also is the default to print, so 'print;' should suffice in this case.

Re: grep: not enough memory ???

harry d brown jr — Thu, 28 Feb 2002 13:56:51 GMT

Richard,

I was just goinf to suggest you getting GNU's grep, but procura beat me to it. Like stated, it will do block scans. The issue is also that you are trying to "display" the "block" of data containing your "string", and if the start of this block (last NewLine) was a few meg's ago (and let's not forget where the block ends - yes, at the next NewLine), you are going to get a lot of crap on your screen.

live free or die
harry