grep: not enough memory ???

[someone_4]

Hello everyone ..
seems I am starting eat live and dream HPUX .. fun huh ..
Well we are trying to cat and grep a huge file.
154082574 Feb 26 21:50 radius.log
When I try to grep .. I get this error
cat radius.log | grep spr8345409a1
grep: not enough memory

I can grep a smaller file but I cant get that big file. Does grep have limits? Any other way to look for a string in a huge file?

Thanks,
Richard

[Steven Sim Kok Leong]

Hi,

Try this perl script:

#!/usr/bin/perl

open (FILE, "radius.log");

while ()
{
if /spr8345409a1/
{
print "$_\n";
}
}

Btw, what is your maxdsiz and ulimit?

Hope this helps. Regards.

Steven Sim Kok Leong

[someone_4]

maxdsiz
0x04000000
Calculated Value: 67108864

ulimit -d
65536

Do I need to bump ulimit?
And will a cat file | grep whatever
use allot of cpu ?

Richard

[Steven Sim Kok Leong]

Hi,

Does the perl script work for you? If it worked, then it was probably a grep limitation.

Hope this helps. Regards.

Steven Sim Kok Leong

[SHABU KHAN]

Hi,

I would first try:
grep "spr8345409a1" radius.log

If this comes back with "not enough memory" then try feeding it to xargs like this:

ls radius.log|xargs grep "spr8345409a1"

Thanks,
Shabu

[someone_4]

no the perl script spit out some errors.

syntax error at ./perl.scr line 5, near "if /spr8345409a1/"
syntax error at ./perl.scr line 9, near "}"
Execution of ./perl.scr aborted due to compilation errors.

Richard

[Steve Steel]

Hi


This could be swap so check your syslog.


Also try the old do loop to slow it down


cat file|while read line
do
echo $line|grep spr8345409a1
done


Steve Steel

[Rodney Hills]

To fix perl script, change "if" to-

if ( /spr8345409a1/ )

-- Rod Hills

[someone_4]

well ..
Here is what the perl script gave me ..

./perl.scr
Out of memory during "large" request for 67112960 bytes at ./perl.scr line 3.

[Rodney Hills]

This out of memory condition might be caused by no ^j (new line) in the file. grep and perl are record processing utilities.

If this is the case, you can use perl to do binary reads, reading in a block at a time and scanning that block.

-- Rod Hills

[Eric Ladner]

You're maxdsize is pretty small (still at the default, I believe), although it seems strange that grep would be causing those kinds of problems.

Is this a binary file? Grep is suppposed to be line oriented and I know I have grepped things out of files in the 1 to 2 gig range, but they were text files with newline characters embedded.

[Frank Slootweg]

If this is on HP-UX 10.20, then there are (customer) reports that grep patch PHCO_13451 *might* help. However your scenario is not specifically mentioned in the .txt file of the patch.

Other reports indicate that maxdsiz should be greater than the file's size, which, as far as I can tell is not the case in your case.

Yet other reports use split(1) to split the file into smaller pieces and grep the pieces. May be you can use this trick to see if maxdsiz is indeed the likely culprit.

Just to be sure: This is a *text* file, right? I.e. a file with lines ending with a newline character and with line lengths less than 2048 bytes (see glossary(9) and limits(5)). If not,you may want to use fold(1).

[H.Merijn Brand (procura]

http://hpux.tn.tudelft.nl/hppd/hpux/Gnu/grep-2.5a/ is the GNU version of grep, which might help you out.

With all those memory requirements passing by, a question might be apropriate: Are the `lines' in radius.log separated by newlines, or by something else? If there are no newlines, most unix utilities will fail, because they work line based.

GNU grep works block based, and perl works line based by default, but can easily be told to work block based or accept any record separator you like.

with perl you might get better results in setting the input record separator to something /not/ in the pattern. But that still leaves the question: what should be printed if the lines are not newline separated?

perl -ne '/spr8345409a1/ and print' radius.log

for a much simpler version than all perl examples given, and a safer version could be


Steven's version has some bugs (apart from the missing () in the if):

#!/usr/bin/perl

open (FILE, "radius.log");

while () {
if (/spr8345409a1/) {
print "$_\n";
}
}

1. open FILE, "< radius.log" or die "radius.log: $!";
always check the return status of the open, reading from a not opened handle might be surprising
always open files for reading using "<" in front to prevent strange and unwanted effects:

my $x = "radius.log"
open LOGFILE, $x;

care to think what would happen if $x came from the outside world and contains "| xargs rm -rf /"?

2. print;

perl reads /lines/ separated by newlines (at least on unix) but does not strip them, so $_ still has the newline at the end. $_ also is the default to print, so 'print;' should suffice in this case.

[harry d brown jr]

Richard,

I was just goinf to suggest you getting GNU's grep, but procura beat me to it. Like stated, it will do block scans. The issue is also that you are trying to "display" the "block" of data containing your "string", and if the start of this block (last NewLine) was a few meg's ago (and let's not forget where the block ends - yes, at the next NewLine), you are going to get a lot of crap on your screen.

live free or die
harry