What script to use

[Kenneth_18]

Hello,

I am working with a huge log file that is space/tab delimeted like below:

AAABBB CCCDDD EEEFFF GGGHHH JJJKKK
LLLMMM NNNOOO PPPQQQ RCCCSS TTTUUU
ZCCCXX YYYHHH FFFLLL SDFAWE SWRDFG
QWEsdf CCCsdh asderi asdkef lksdoj

What I want to accomplish is after the log file has gone to the script, the script would only display or list the line which contains a pattern in a specific column.

So for the example above, I only want to list or display the lines which contains the "CCC" pattern on column 2. So the output file of the script would be:

AAABBB CCCDDD EEEFFF GGGHHH JJJKKK
QWEsdf CCCsdh asderi asdkef lksdoj

What I usually do is import this textfile into MS excel and sort it but MS excel can handle only upto 65K rows whereas the text file I'm using is up to a 100K rows or more. Also, I wish to do this in a unix enviroment and not transfer from one system to another.

Which editor can handle this better AWK or SED and could you give me a sample working script.

Any help is appreciated as I'm totally not very familiar with SED or AWK or scripting in general.

Thanks in advance.

Kenneth

[Balaji N · Accepted solution]

see if this helps.

+++++
cat a
AAABBB CCCDDD EEEFFF GGGHHH JJJKKK
LLLMMM NNNOOO PPPQQQ RCCCSS TTTUUU
ZCCCXX YYYHHH FFFLLL SDFAWE SWRDFG
QWEsdf CCCsdh asderi asdkef lksdoj
[balajin@ec2t5101743 tmp]$ awk ' $2 ~ /CCC/ { print }' a
AAABBB CCCDDD EEEFFF GGGHHH JJJKKK
QWEsdf CCCsdh asderi asdkef lksdoj
+++++++++++++++++++++++++


$2 is the field you want to search for the and with in the slashes you put the characters you want to search for.

hth
-balaji

[David_246]

Hi Kenneth,

This is exactly where you have logsurfer for.

You have a log-file of a specific size.
You create a conf-file with search patterns and actions.

So in your case lines of :

'^([^ ]+) CCC*' - - - 0 pipe '/usr/local/bin/myprogram "Clines"'
'^([^ ]+) BBB([^ ]+) CCC*' - - - 0 pipe '/usr/local/bin/myprogram "BandClines"'

The first rule will start with a column of unknown characters, than the secons column has only CCC and followed by more columns.
The secons rule will start with a column of unkown characters and followed by BBBsomethingelse and the third column CCCand some other characters and/or columns.
You create the script myprogram that reads the first param and adds it to your file. By using a program to call you can do lot's of manipulation, etc and you can re-use it every time.

I'll add all the files for you of logsurfer, inclusive the ./logsurfer/etc/conf.syslog with lots of examples !

Regs David

[Steve Steel]

Hi

file=$1
typeset -i column=$2
search=$3
cat $file|while read line
do
set $(echo $line)
if [ "$column" -gt "1" ]
then
typeset -i colum1=$column-1
shift $colum1
fi
y=$1
x=${y#$search}
if [ "$x" != "$1" ]
then
echo $line
fi
done


Parameters are fiel column and pattern


Steve Steel

[Wilfred Chau_1]

cat |awk '$2 ~/^CCC/ {print}'

[Sridhar Bhaskarla]

Hi Kenneth,

awk '$2 ~ "^CCC" {print}' your_file

-Sri