Search entire system for word(s) in all text files

[William Pribble]

Hello,

I need to search my entire system for a number of key words. I only want to search text files. When I run the following script, it runs for a while, but errors out with grep running out of memory.
I separated the grep statements into a few key words each statement.
Any help would be greatly appreciated.

#!/usr/bin/ksh
set -x


> /tmp/out_file1

cd /
for mp in `find . -type d ! -name . -prune | sed -e "s/^..//"`
do


for i in `find /${mp} -type f | xargs file | egrep 'commands|script' | awk -F
'{print $1}'`
do

QUESTION=FALSE

egrep 'devservice|h50app|devpriv' $i > /dev/null
if [ "$?" -eq 0 ]
then
QUESTION=TRUE
fi
egrep 'appswervice|apppriv|s70app' $i > /dev/null
if [ "$?" -eq 0 ]
then
QUESTION=TRUE
fi

egrep 'dbservice|sbpriv|s70db' $i > /dev/null
if [ "$?" -eq 0 ]
then
QUESTION=TRUE
fi

egrep 'syncservic|h50sync|uname' $i > /dev/null
if [ "$?" -eq 0 ]
then
QUESTION=TRUE
fi

egrep 'ftp|rcp|rsh' $i > /dev/null
if [ "$?" -eq 0 ]
then
QUESTION=TRUE
fi

#egrep 'devservice|h50app|devpriv|appservice|apppriv|s70app|dbservice|sbpriv|s70
db|syncservic|h50sync|uname|ftp|rcp|rsh' $i > /dev/null

if [ "$QUESTION" = "TRUE" ]
then
echo $i >> /tmp/out_file1
fi

done

done

[James R. Ferguson]

Hi Elaine:

A few comments:

Instead of 'grep'ing for your regular expressions and then invoking 'awk' to print the first field, create a single 'awk' script to do both.

The "for X in `command`" sequences are consuming memory constructing an argument list. You should consider building a temporary file of arguments to be read as input, or use a pipeline.

Regards!

...JRF...

[William Pribble]

Hi James,

Thanks for the pointers. How do you make a pipeline for the commands, or can you point me to a resource to find the information.

Thanks again

[James R. Ferguson]

Hi (again):

Consider:

#!/usr/bin/sh
find /tmp -type f -name "*.log" |
while read LINE
do
echo "processing ${LINE}"
done
exit 0

Regards!

...JRF...

[H.Merijn Brand (procura · Accepted solution]

# perl -MFile::Find -le'$/=undef;find(sub{-f&&-T or return;local@ARGV=($_);<>=~/pattern/ and print$File::Find::name},"/")'

Enjoy, have FUN! H.Merijn

[William Pribble]

H.Merijn,

Thank you for your response. Because I am not familiar with perl... do you mind go over how your script functions for me. Thanks again for all your help.

[H.Merijn Brand (procura]

?? Invoke perl

# perl

?? use standard module File::Find

-MFile::Find

?? end all print's in perl with a newline

-l

?? and process the following expression

-e

The expression broke down in multiple lines:

?? Undefine the input record seprarator (default newline), so a read on the file causes the file to be read completely

$/ = undef;

?? Call the find function (use 'man File::Find for the specifics)

find (

?? Pass it two arguments: a subroutine reference to be called on every entry found, and a list of folders to search

sub {

?? I only want *files* (not pipes, folders/directories, or special devices) that are *plain text (not binary), as your original quest states

-f && -T or return;

?? magic open :) put the filename teporary in the arg list, so <> (the read operator) reads from the current file without having to explicitely open it. Very efficient. Remember that <> will read the complete file in one single string, because we undef'ed the input record separator

local @ARGV= ($_);

?? Read one line (the complete file in our case) and match it to pattern

<> =~ /pattern/

It might actually have been better to write this as /pattern/s, but that depends on what kind of pattern you use (see perldoc perlre)

?? If it matches, print the full name of the file it currenty parses, followed by a newline (remember the -l option)

and print $File::Find::name

?? end of closure (subroutine ref)

}

?? and the list of folders/directories to scan

, "/"

)'

HTH Enjoy, have FUN! H.Merijn

[Graham Cameron_1]

Put your keywords into a file, say "kfile", and use:

nice find / -type f | xargs grep -il -f kfile

The -i option on grep means ignore case, you may not need this.

Note also that this will search any NFS mounted filesystems, you may not want this. To avoid, see my post in this thread:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x015bcf29a397ca4db25109ab3743467e,00.html

[Tony Constantine]

or

for i in file1 file2 file3
do
echo "searching for $i"
find . -type f -exec grep -il $i {} \;

[Steve Post]

Not exactly related to your problem but....
fyi about looping. I know of three ways.
1.
for I in $List
do
echo $I
done

2.
I=10
while [ $I -gt 0 ]
do
echo $I
I=`expr $I -1`
done

3. and here is the file loop.
exec 3while read Line 0<&3
do
echo $Line
done
3<&-

steve

[Bhuvaneswari Selvaraj]

Create a file containing only the key words that you
want t o search and name it as keyfile.

for i in `cat keyfile`
do
find . -name "*.txt" -exec grep $i {} \; -print 2>/dev/null
done

If you want to search all regular files then

for i in `cat keyfile`
do
find . -type f -exec grep $i {} \; -print 2>/dev/null
done

if you still want to save the no of finds,
find . -type f > filelist
for i in `cat filelist`
do
for j in `cat keyfile`
do
grep $j $i
done
done

Depending on what you want as output, use the return value of grep and print accordingly

[john korterman]

Hi,
try changing the line:
`for i in `find ./${mp} -type f | xargs file |egrep 'commands|script' | awk -F '{print $1}'`

to:
`for i in `find ./${mp} -type f | xargs file | awk -F: '$2 ~ /text$/ {print $1}'`

regards,
John K.

[hein coulier]

here's on i use :

#!/bin/ksh

DIR=$1
TEXT=$2

find $DIR -type f -exec file {} \;|
grep text|
while read line
do
fname=`echo $line|cut -f1 -d':'`
grep "$TEXT" "$fname" /dev/null
done

[William Pribble]

H. Merijn,
Thanks for the break down on the perl script.
I ran the line and it does seem to work, only I am getting an "out of memory" error shortly into the run. Any ideas or thoughts on this.
Thanks again for all your help.

[H.Merijn Brand (procura]

What perl do you use? (perl -v)

And don't forget to award all those fine answers above with points. Bear to heart that none of the above answers will work correctly with NFS (well, some might behave, but don't expect any speed :)

Enjoy, have FUN! H.Merijn

[William Pribble]

Perl version 5.005_03

[H.Merijn Brand (procura]

Any special reason why you're still using tha err old version? It's an OK version, and it's very stable, but you'll miss all the new fun features.

OTOH I think that my snippet should run OK on any perl5.

FWIW if you wanna upgrade, I've got a bunch of precompiled binaries laying around on https://www.beepz.com/personal/merijn/ or http://www.cmve.net/~merijn/

Enjoy, have FUN! H.Merijn